Service providing apparatus and method, and information processing apparatus and method

ABSTRACT

Systems and methods are provided which can be used to control electrical household appliances connected to a home network by use of a GUI having an entertainment-oriented virtual space capability. A virtual space is arranged with a mascot dedicated to the user of an information processing apparatus. An icon can be clicked to control a real world device existing in the home of a user. A user can participate in a virtual space belonging to another user through sharing.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.10/498,582, filed on Feb. 17, 2005, which is a is a national stageapplication under 35 U.S.C. §371 of International Application No.PCT/JP2003/012857, filed Oct. 8, 2003, which claims priority fromJapanese Application No. P2002-297386, filed Oct. 10, 2002, thedisclosures of which are hereby incorporated by reference herein.

BACKGROUND OF THE. INVENTION

1. Technical Field

The present invention relates generally to an information processingsystem, a service providing apparatus and method, an informationprocessing apparatus and method, a recording medium, and a program and,more particularly, to an information processing system, a serviceproviding apparatus and method, an information processing apparatus andmethod, a recording medium, and a program for use in remotelycontrolling real devices by operating icons in a virtual space.

2. Background Art

A variety of methods have been proposed for remotely controllingelectrical household appliances.

For example, Japanese Patent laid-open No. 2002-44765 discloses a systemcontrolling, via the Internet, the electrical household appliancesconnected to a home network. For example, Japanese Patent No. 3016350discloses an application program for personal computers using an agentfor the interface for the remote control of electrical householdappliances.

For the programs for remotely controlling electrical householdappliances by using personal computers via a network, UPnP, Jini, andHAVi are known, for example. These programs are middleware, and the userinterface, which is operated by users for controlling their electricalhousehold appliances, depends on other software programs using themiddleware such as UPnP.

On the other hand, a method is disclosed in Japanese Patent Laid-openNo. 2001-154966, for example, in which a plurality of userssimultaneously participate in a virtual space built on a computernetwork represented by the Internet, thereby realizing realtimecommunication.

These software programs which provide user interfaces for the remotecontrol of electrical household appliances have only the functions ofcontrolling electrical household appliances, namely, practicality, inmany cases. Therefore, such software programs present a problem thatthey cannot satisfy the needs of users who demand not only practicalitybut also amusement on the software programs of their personal computers.

Technologies which enable communication within a virtual spaces alsopresent a problem of lack of close communication between users ascommunication is achieved by users' entering public, not private,spaces.

SUMMARY OF THE INVENTION

An embodiment of the present invention can be used to control theelectrical household appliances connected to a home network by use ofthe GUI (Graphical User Interface) having an amusement virtual space andto form a private shared space in the virtual space, thereby providingcloser communication between users.

According to an aspect of the present invention, there is provided aninformation processing system having a service providing apparatusincluding providing means for providing, to each of the plurality ofinformation processing apparatuses, a service of a private virtual spacein which a mascot is arranged; storage means for storing a group towhich a user of each of the plurality of information processingapparatuses belongs; sharing means for sharing a service of a privatevirtual space provided for the information processing apparatus used bythe user, the virtual space containing a mascot dedicated to an otheruser belonging to the group, with the information processing apparatusused by the other user belonging to the group; and relay means forrelaying data realtime between the plurality of information processingapparatuses used by the users belonging to the group; each of theplurality of information processing apparatuses including request meansfor requesting the participation of a mascot of said informationprocessing apparatus into a private space provided for an otherinformation processing apparatus; display control means for controllingthe displaying of the virtual space provided by the service providingapparatus in response to the request; and communication means forcommunicating data realtime with the other information processingapparatus via the service providing apparatus.

The above-mentioned storage means of the service providing apparatusstores the group to which the user belongs, by use of identificationinformation of the user or identification information of the informationprocessing apparatus.

The above-mentioned storage means of the service providing apparatusstores the group to which the user belongs for each of the privatevirtual spaces provided for the user and the other user. Theabove-mentioned data may be text data. The above-mentioned data may beimage data.

The above-mentioned information processing apparatus may further includereproduction means for reproducing stream data received via the network;and notification means for notifying the other information processingapparatus that the reproduction means has started the reproduction ofthe stream data.

The above-mentioned reproduction means, if notified by the otherinformation processing apparatus that the reproduction of the stream hasstarted, may receive the stream data distributed via the network andreproduces the stream data.

The above-mentioned information processing apparatus may further includecontrol means for controlling a really existing electronic device viathe network by operating an icon corresponding to an electronic devicearranged in the virtual space.

According to an aspect of the present invention, there is provided aservice providing apparatus including providing means for providing aservice of a private virtual space in which a mascot is arranged to eachof the plurality of information processing apparatuses; storage meansfor storing a group to which a user of each of the plurality ofinformation processing apparatuses belongs; sharing means for sharing aservice of a private virtual space provided for the informationprocessing apparatus used by the user, the virtual space containing amascot dedicated to an other user belonging to the group, with theinformation processing apparatus used by the other user belonging to thegroup; and relay means for relaying data realtime between the pluralityof information processing apparatuses used by the users belonging to thegroup.

According to an aspect of the present invention, there is provided aservice providing method including the steps of providing a service of aprivate virtual space in which a mascot is arranged to each of theplurality of information processing apparatuses; storing a group towhich a user of each of the plurality of information processingapparatuses belongs; sharing a service of a private virtual spaceprovided for the information processing apparatus used by the user, thevirtual space containing a mascot dedicated to an other user belongingto the group, with the information processing apparatus used by theother user belonging to the group; and relaying data realtime betweenthe plurality of information processing apparatuses used by the usersbelonging to the group.

According to an aspect of the present invention, there is provided afirst recording medium including the steps of providing a service of aprivate virtual space in which a mascot is arranged to each of theplurality of information processing apparatuses; storing a group towhich a user of each of the plurality of information processingapparatuses belongs; sharing a service of a private virtual spaceprovided for the information processing apparatus used by the user, thevirtual space containing a mascot dedicated to an other user belongingto the group, with the information processing apparatus used by theother user belonging to the group; and relaying data realtime betweenthe plurality of information processing apparatuses used by the usersbelonging to the group.

According to an aspect of the present invention, there is provided afirst program executing the steps of providing a service of a privatevirtual space in which a mascot is arranged to each of the plurality ofinformation processing apparatuses; storing a group to which a user ofeach of the plurality of information processing apparatuses belongs;sharing a service of a private virtual space provided for theinformation processing apparatus used by the user, the virtual spacecontaining a mascot dedicated to an other user belonging to the group,with the information processing apparatus used by the other userbelonging to the group; and relaying data realtime between the pluralityof information processing apparatuses used by the users belonging to thegroup.

The above-mentioned storage step may store the group to which the userbelongs, by use of identification information of the user oridentification information of the information processing apparatus.

The above-mentioned storage step may store the group to which the userbelongs for each of the private virtual spaces provided for the user andthe other user. The above-mentioned data may be text data. Theabove-mentioned data may be image data.

According to an aspect of the present invention, there is provided aninformation processing apparatus including request means for requestingthe participation of a mascot of the information processing apparatusinto a private virtual space provided for an other informationprocessing apparatus; display control means for controlling thedisplaying of the virtual space provided from the service providingapparatus upon the request; and communication means for communicatingdata realtime with the other information processing apparatus via theservice providing apparatus.

According to an aspect of the present invention, there is provided aninformation processing method including the steps of requesting theparticipation of a mascot of the information processing apparatus into aprivate virtual space provided for an other information processingapparatus; controlling the displaying of the virtual space provided fromthe service providing apparatus upon the request; and communicating datarealtime with the other information processing apparatus via the serviceproviding apparatus.

According to an aspect of the present invention, there is provided asecond recording medium including the steps of requesting theparticipation of a mascot of the information processing apparatus into aprivate virtual space provided for an other information processingapparatus; controlling the displaying of the virtual space provided fromthe service providing apparatus upon the request; and communicating datarealtime with the other information processing apparatus via the serviceproviding apparatus.

According to an aspect of the present invention, there is provided asecond program executing the steps of requesting the participation of amascot of the information processing apparatus into a private virtualspace provided for an other information processing apparatus;controlling the displaying of the virtual space provided from theservice providing apparatus upon the request; and communicating datarealtime with the other information processing apparatus via the serviceproviding apparatus.

The above-mentioned data may be text data. The above-mentioned data maybe image data.

The above-mentioned second program may further include the steps ofreceiving stream data distributed via the network and reproducing thestream data, and notifying the other information processing apparatusthat the reproduction of the stream data has started.

The process of the reproduction step, if notified by the otherinformation processing apparatus that the reproduction of the stream hasstarted, may receive the stream data via the network and reproduces thestream data.

The above-mentioned second program may further include the step ofcontrolling a really existing electronic device via the network byoperating an icon corresponding to an electronic device arranged in thevirtual space.

In the information processing system according to an aspect of thepresent invention, a group to which the users of information processingapparatuses belong is stored in the service providing apparatus; theservice of a virtual space, which is a private virtual space providedfor each of information processing apparatuses used by users andincludes a mascot dedicated to an other user belonging to the samegroup, is shared by the information processing apparatus used by thisother user; and data are relayed realtime between the informationprocessing apparatuses used by two or more users belonging to the samegroup. In addition, one information processing apparatus requests forthe participation of its mascot in the private virtual space providedfor an other information processing apparatus; in response to thisrequest, the displaying of the virtual space provided by the serviceproviding apparatus is controlled; and data are communicated realtimewith this other information processing apparatus via the serviceproviding apparatus.

In the service providing apparatus and method and program according toan aspect of the present invention, a group to which the users ofinformation processing apparatuses belong is stored in the serviceproviding apparatus; the service of a virtual space, which is a privatevirtual space provided for each of information processing apparatusesused by users and includes a mascot dedicated to an other user belongingto the same group, is shared by the information processing apparatusused by this other user; and data are relayed realtime between theinformation processing apparatuses used by two or more users belongingto the same group.

In the information processing apparatus and method and program accordingto an aspect of the present invention, one information processingapparatus requests for the participation of its mascot in the privatevirtual space provided for an other information processing apparatus; inresponse to this request, the displaying of the virtual space providedby the service providing apparatus is controlled; and data arecommunicated realtime with this other information processing apparatusvia the service providing apparatus.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating an exemplary configuration ofa virtual space remote control operation system to which the presentinvention is applied.

FIG. 2 is a block diagram illustrating an exemplary configuration of aportable user terminal shown in FIG. 1.

FIG. 3 is a block diagram illustrating an exemplary configuration of aPC shown in FIG. 1.

FIG. 4 is a functional block diagram of the PC shown in FIG. 1.

FIG. 5 illustrates an exemplary display of a virtual home viewer window.

FIG. 6 illustrates icons arranged in a virtual room.

FIG. 7 is a flowchart for describing login processing.

FIG. 8 illustrates an exemplary display of a virtual home viewer windowto be displayed upon login.

FIG. 9 is a flowchart for describing layout information acquisitionprocessing.

FIG. 10 illustrates an exemplary display of a virtual room.

FIG. 11 is a flowchart for describing event information acquisitionprocessing.

FIG. 12 is a flowchart for describing home network informationacquisition processing.

FIG. 13 is a flowchart for describing member information acquisitionprocessing.

FIG. 14 is a flowchart for describing logout processing.

FIG. 15 illustrates an exemplary display upon selection of a door icon.

FIG. 16 illustrates the movement of an icon.

FIG. 17 illustrates the movement of another icon.

FIG. 18 illustrates the change in the orientation of an icon.

FIG. 19 illustrates an operation for arranging an icon by superimposingthe icon on another icon.

FIG. 20A illustrates a difference between an operation for arranging anicon by superimposing the icon on another icon and an operation forarranging an icon behind another icon.

FIG. 20B also illustrates a difference between an operation forarranging an icon by superimposing the icon on another icon and anoperation for arranging an icon behind another icon.

FIG. 21 illustrates the storage of an icon in a closet.

FIG. 22 illustrates an exemplary display upon selection of a closeticon.

FIG. 23 illustrates an exemplary display of an item list.

FIG. 24 illustrates an exemplary display in which one item has beenselected from the item list.

FIG. 25 illustrates an operation for getting an icon from the closetinto the room.

FIG. 26 also illustrates an operation for getting an icon from thecloset into the room.

FIG. 27 illustrates an exemplary display upon selection of a PC icon.

FIG. 28 illustrates an exemplary display in which a video recorder icon,an audio player icon, and a photo album icon have been separated fromthe PC icon.

FIG. 29 illustrates an exemplary display in which the video recordericon has been separated.

FIG. 30 illustrates an exemplary display of a television program guide.

FIG. 31 is a flowchart for describing EPG information acquisitionprocessing.

FIG. 32 illustrates an exemplary display upon selection of thetelevision program guide.

FIG. 33 illustrates a recording operation.

FIG. 34 illustrates an exemplary display in which the video recordericon is selected.

FIG. 35 illustrates an exemplary display of a recorded program list.

FIG. 36 is a flowchart for describing content list informationacquisition processing.

FIG. 37 illustrates an exemplary display upon selection of one ofrecorded programs.

FIG. 38 illustrates an operation for selecting a destination to whichthe reproduction of a recorded program is outputted.

FIG. 39 illustrates an exemplary display when the video of the recordedprogram is displayed on a main panel.

FIG. 40 is a flowchart for describing the processing of displaying thevideo of the recorded program on the main panel.

FIG. 41 illustrates an exemplary display of the main panel when thevideo reproduced from the recorded program is displayed on a real TVset.

FIG. 42 is another exemplary display illustrating the main panel whenthe video reproduced from the recorded program is displayed on the realTV set.

FIG. 43 is a flowchart for describing the processing of displaying thevideo of the recorded program on the real TV set.

FIG. 44 illustrates an exemplary display upon selection of a TV icon.

FIG. 45 illustrates an operation for controlling the real TV set.

FIG. 46 illustrates another operation for controlling the real TV.

FIG. 47 illustrates an exemplary display upon selection of a notepadicon.

FIG. 48 illustrates an exemplary display upon pressing of “New” buttonshown in FIG. 47.

FIG. 49 illustrates an exemplary display when text is written to a pageof a notepad.

FIG. 50 illustrates an exemplary display upon selection of a memo sheeticon.

FIG. 51 illustrates an exemplary display upon selection of a telephoneicon.

FIG. 52 illustrates an exemplary display upon pressing of “Member List”button shown in FIG. 51.

FIG. 53 illustrates an exemplary display upon pressing of “User Search”button shown in FIG. 52.

FIG. 54 illustrates an operation for user search.

FIG. 55 illustrates an operation for provisionally registering “Nanako”with a member list.

FIG. 56 illustrates an exemplary display upon provisional registrationof “Nanako”.

FIG. 57 illustrates an operation for deleting “Hideo” from the memberlist.

FIG. 58 is an exemplary display after deletion of “Hideo” from themember list.

FIG. 59 illustrates an exemplary display of a virtual room ofprovisionally registered “Nanako”.

FIG. 60 is a flowchart for describing the processing of provisionalregistration.

FIG. 61 illustrates an exemplary display upon selection of the telephoneicon.

FIG. 62 illustrates an operation for making a call to “Yumiko”.

FIG. 63 is a flowchart for describing current status informationacquisition processing.

FIG. 64 is a flowchart for describing telephone chat processing.

FIG. 65 illustrates an exemplary display when a call has come.

FIG. 66 illustrates an exemplary display showing that a talk is beingmade.

FIG. 67 illustrates an operation for a telephone chat.

FIG. 68 illustrates an operation for ending a telephone chat.

FIG. 69 is a flowchart for describing telephone chat end processing.

FIG. 70 is flowchart for describing recorded message recordingprocessing.

FIG. 71 illustrates an exemplary display if a called party is anautomatic answering machine.

FIG. 72 illustrates an exemplary display upon selection of the telephoneicon.

FIG. 73 is an exemplary display of a recorded message list.

FIG. 74 illustrates an operation for making a confirmation of recordedmessages.

FIG. 75 is an exemplary display upon selection of a door icon.

FIG. 76 illustrates an operation for visiting “Yumiko's Home”.

FIG. 77 is a flowchart for describing conference generation processing.

FIG. 78 illustrates an exemplary display upon visiting “Yumiko's Home”.

FIG. 79 is a flowchart for describing the processing of sharinginformation corresponding to user B's operation as conferencesynchronous information.

FIG. 80 illustrates an operation for calling through an interphone.

FIG. 81 is a flowchart for describing the processing of sharinginformation corresponding to user A's operation as conferencesynchronous information.

FIG. 82 illustrates an exemplary display of a virtual room correspondingto that a “Chime” button has been pressed.

FIG. 83 is a flowchart for describing interphone chat processing.

FIG. 84 illustrates an exemplary display at the time of interphone chat.

FIG. 85 illustrates an operation for ending interphone chat.

FIG. 86 illustrates an exemplary display at the time of interphone chat.

FIG. 87 illustrates an operation for unlocking the door.

FIG. 88 illustrates an operation for entering the room.

FIG. 89 illustrates an exemplary display when mascot A enters thevirtual room of user B.

FIG. 90 illustrates another exemplary display when mascot A enters thevirtual room of user B.

FIG. 91 is a flowchart for describing the processing from unlocking thedoor to the entering of the room.

FIG. 92 illustrates a method of directly passing an item betweenmascots.

FIG. 93 illustrates a method of directly passing an item betweenmascots.

FIG. 94 illustrates a method of directly passing an item betweenmascots.

FIG. 95 is a flowchart for describing direct ownership processing.

FIG. 96 illustrates an operation for confirming a directly passed item.

FIG. 97 illustrates an operation for arranging the directly passed itemin the virtual room.

FIG. 98 illustrates a method of indirectly passing an item via amailbox.

FIG. 99 illustrates a method of indirectly passing the item via themailbox.

FIG. 100 illustrates an operation for confirming the item indirectlypassed via the mailbox.

FIG. 101 illustrates an operation for arranging the item directly passedvia the mailbox into the virtual room.

FIG. 102 is a flowchart for describing indirect ownership processing.

FIG. 103 illustrates an operation for Internet radio sharing processing.

FIG. 104 illustrates an operation for Internet radio sharing processing.

FIG. 105 is a flowchart for describing Internet radio sharingprocessing.

FIG. 106 illustrates an operation for image sharing processing.

FIG. 107 illustrates an exemplary display at the time of image sharingprocessing.

FIG. 108 illustrates an operation for image sharing processing.

FIG. 109 illustrates an operation for image sharing processing.

FIG. 110 is a flowchart for describing image sharing processing.

FIG. 111 illustrates an exemplary display of a shared image.

FIG. 112 is a flowchart for describing image sharing end processing.

DETAILED DESCRIPTION Best Mode for Carrying Out the Invention

The following describes a virtual space remote control system accordingto the present invention by way of example with reference to theaccompanying drawings. This virtual space remote control system allowseach user to view his or her own virtual space (hereafter also referredto as a virtual home) on a portable user terminal as a CG (ComputerGraphics) image and allows each user to operate a computer icon and atelevision set icon, for example, arranged in the virtual home, therebycontrolling a personal computer (hereafter referred to as a PC), atelevision set (hereafter referred to as a TV set), audio equipment, andother Audio/Video (AV) equipment, and various electronic equipment thatexist in the user's real room.

The virtual space remote control system also allows each user to visitthe virtual home of an other user, exchange messages with other users,and share stream data including music and image data with other users.

In what follows, the processing in which a user operates the screen ofthe virtual home displayed on the portable user terminal or the like tocontrol the PC existing in the user's real home without having anyinteraction with any other users is referred to as standaloneprocessing.

On the other hand, the processing in which a plurality of users interacteach other like visiting an other user's home, for example, is referredto as communication processing.

Now, referring to FIG. 1, there is shown an exemplary configuration ofthe virtual space remote control system. This system is divided into theequipment on the user side and the equipment on the service providerside.

First, the equipment on the user side will be described. For example, aportable user terminal 1, including a PC, a PDA (Personal DigitalAssistant), or a mobile telephone and connected to desired servers, forexample, via a base station 4 and the Internet 5 realizes the followingoperations by executing a virtual home viewer program 51 (FIG. 2) by aCPU 31 (FIG. 2).

The portable user terminal 1, which is used by user A, connects to avirtual home DB server 21, for example, on the service provider side viathe base station 4 and the Internet 5 to get the information about thevirtual home of user A and displays its image on the screen. This imageof the virtual home shows a PC icon and a TV set icon corresponding to aPC 12 and a TV set 13, which exist in the real home of user A (hereafterreferred to as user A's home).

In response to a user operation performed on the PC icon or the TV seticon in the virtual home displayed on the screen, the portable userterminal 1 connects to a home server 11 included in a home network 2 ofuser A's home via the base station 4 and the Internet 5, thereby sendingcontrol signals for controlling the PC 12 and the TV set 13.

User A's home has the home network 2 arranged around the home server 11.The home server 11 has a global IP address. Therefore, the portable userterminal 1 or the like can access the home server 11 via the Internet 5.From the portable user terminal 1 via the Internet 5, the home server 11receives control signals for controlling the PC 12 and the TV set 13 andexecutes the middleware such as UPnP, thereby passing the receivedcontrol signals to the PC 12 and the TV set 13 via a data bus 14.

As with the portable user terminal 1, the PC 12 of user A's homeconnects to the virtual home DB server 21, for example, on the serviceprovider side via the Internet 5, thereby displaying the user A'svirtual home on the screen. Also, in response to user's operationperformed on the PC icon or the TV set icon in the displayed virtualhome, the PC 12 sends control signals for controlling the TV set 13, forexample, connected to the home server 11.

As with the portable user terminal 1, the PC 3, which is used by user B,connects to the virtual home DB server 21, for example, on the serviceprovider side to display the user B's virtual home on the screen. Also,in response to user's operation performed on the PC icon or the TV seticon in the displayed virtual home of user B, the PC 3 connects to thehome server of user B's home via the Internet 5 to send control signalsfor controlling its PC (not shown) or the like connected to its homeserver (not shown).

It should be noted that the user-side equipment includes the portableuser terminals and PCs being operated by users other than user A anduser B and there are also home servers for these users other than user Aand user B, the illustrations thereof being omitted for the brevity ofdescription.

The following describes the equipment on the service provider side. Thevirtual home DB server 21 stores virtual home layout information (theexternal view, wall pattern, and floor material of the virtual home andthe icon arrangement in the virtual home, for example), which is set byeach user in a desired manner, and each user's event information (theinformation telling that messages addressed to that user are in themailbox, the information indicative of applications for chat by otherusers, and so on). The virtual home DB server 21 provides these piecesof information to the portable user terminal 1, for example, whichaccesses the virtual home DB server 21 via the Internet 5.

An authentication server 22 executes authentication processing on theportable user terminal 1 or the like, which has accessed the equipmenton the service provider side via the Internet 5. To be more specific,the authentication server 22 provides the access right for the equipmenton the service provider side and determines the validity or invalidityof the access right to be sent to the virtual home DB server 21, aninter-user-terminal communication relay server 23, or a user informationholding server 24.

The inter-user-terminal communication relay server 23 executes theprocessing associated with a plurality of users who use the virtualspace remote control system. For example, the inter-user-terminalcommunication relay server 23 stores a member list of a user group towhich each user belongs, manages the chat being executed between aplurality of users, and manages the conference synchronous information(details of which will be described later) being generated when avirtual room is shared.

It should be noted that each user can belong to a plurality of usergroups.

The user information holding server 24 stores the personal data of eachuser; namely, the event detail information (mail addressed to that userand message text data, for example) for event information, a variety ofcertificate data (details of which will be described later) obtained byeach user, the text data written by the user by utilizing notepadcapability, and the image data to be shared by other users, for example.

In response to a request by the portable user terminal 1, for example,which has accessed via the Internet 5, an EPG (Electronic Program Guide)server 25 provides EPG information including a television program guideand detail information about each program.

An audio server 26 is a broadcasting server of so-called Internet radioand provides audio stream data in response to a request by the portableuser terminal 1 or the like, which has access via the Internet 5.

It should be noted that the equipment devices on the service providerside, the virtual home DB server 21 through the audio server 26, neednot be arranged in a separate manner as shown; some of the servers maybe integrated into one unit if required.

Referring to FIG. 2, there is shown an exemplary configuration of theportable user terminal 1. The portable user terminal 1 incorporates theCPU (Central Processing Unit) 31. The CPU 31 is connected to aninput/output interface 35 via a bus 34. The bus 34 is connected to a ROM(Read Only Memory) 32 and a RAM (Random Access Memory) 33.

The input/output interface 35 is connected to an output block 36 foroutputting sound or the like, an operator input block 37 based on aninput device such as a keyboard and a touch panel for inputting operatorcommands by the user A, a display block 38 based on CRT (Cathode RayTube), LCD (Liquid Crystal Display), or the like on which images such asvirtual home are shown, a storage block 39 based on a hard disk drive orthe like for storing various programs and data, and a communicationblock 40 for performing communication via the base station 4 and theInternet 5. The input/output interface 35 is also connected to a drive41, which reads and writes data on a recording medium such as a magneticdisk 42, an optical disk 43, a magneto-optical disk 44, or asemiconductor memory 45.

The CPU 31 executes various processing operations to be described laterin accordance with the virtual home viewer program 51 loaded from thestorage block 39 into the RAM 33, this program being read from theabove-mentioned recording medium into the storage block 39. This is theend of the description for the exemplary configuration of the portableuser terminal 1.

Referring to FIG. 3, there is shown an exemplary configuration of the PC12. The PC 12 incorporates a CPU 61. The 61 is connected to aninput/output interface 65 via a bus 64. The bus 64 is connected to a ROM62 and a RAM 63.

The input/output interface 65 is connected to an output block 66 foroutputting sound or the like, an operator input block 67 based on aninput device such as a keyboard and a mouse for inputting operatorcommands by the user A, a display block 68 based on CRT, LCD, or thelike on which images such as virtual home are shown, a storage block 69based on a hard disk drive or the like for storing various programs anddata, and a communication block 70 for performing communication with thehome server 11 (shown in FIG. 1) and communication via the home server11 and the Internet 5.

The input/output interface 65 is also connected with a drive 73, whichreads and writes data on a recording medium such as a magnetic disk 74,an optical disk 75, a magneto-optical disk 76, or a semiconductor memory77, and a video encoder/decoder 71, which encodes and decodes AV signalsof television broadcast, for example. The video encoder/decoder 71incorporates a tuner 72 for receiving television broadcast.

The CPU 61 executes a virtual home viewer program 81, a HDD (Hard DiskDrive) video recorder 82, an audio data transmission service program 83,and a photograph data transmission service program 84 which may use anyof the recording media such as the magnetic disk 74. The portableterminal 1 may read the recorded data through the semiconductor memory45 (shown in FIG. 2) into the storage block 39 and then into the RAM 33,thereby realizing the four functions shown in FIG. 4, namely, a virtualhome viewer 91, a HDD video recorder 92, an audio data transmissionservice 93, and a photograph data transmission service 94. As describedabove, these programs may be distributed by various recording media orthrough networks such as the Internet 5.

Like the above-mentioned portable user terminal 1, the virtual homeviewer 91 gets the information about the virtual home of user A from thevirtual home DB server 21 via the Internet 5 to display the virtual homeon the screen and, in response to user A's operation performed on the PCicon or the TV set icon in the virtual home, outputs the controlssignals for controlling the PC 12 and the TV set 13 to the home server11. In addition, the virtual home viewer 91 can reproduce so-calledInternet radio as with the audio data transmission service 93 to bedescribed later.

The HDD video recorder 92 controls the video encoder/decoder 71 toencode AV signals of television broadcast or the like and store theresultant coded data into the storage block 69. Also, the HDD videorecorder 92 reads and decodes the coded data from the storage block 69,displays the decoded AV signal video onto the display block 68, andoutputs the sound from the output block 66. Alternatively, the recorder92 supplies the decoded AV signal video to the TV set 13 (shown inFIG. 1) via the data bus 14.

The audio data transmission service 93 gets the steam data of so-calledInternet radio from the audio server 26 and outputs the correspondingsound from the output block 66.

The photograph data transmission service 94 displays the image datastored in the storage block 69 onto the display block 68 or a list ofthumbnail images (contracted images) of plural pieces of image data ontothe display block 68, for example. This is the end of the descriptionfor the exemplary configuration of the PC 12.

It should be noted that an exemplary configuration of the PC 3 issubstantially the same as the exemplary configuration of the portableuser terminal 1 shown in FIG. 2 or the exemplary configuration of the PC12 shown in FIG. 3, so that the description of the exemplaryconfiguration of the PC 3 will be skipped.

The home server 11 and the virtual home DB server 21 through the userinformation holding server 24 are configured in substantially the samemanner as the exemplary configuration of the PC 12 shown in FIG. 3, sothat the description of the exemplary configuration of each of thesecomponents will be skipped. The home server 11 and the virtual home DBserver 21 through the user information holding server 24 execute variousprocessing operations to be described later by which the CPUsincorporated in each server carry out predetermined programs.

Referring to FIG. 5, there is shown an example of a virtual home viewerwindow displayed on the screen of the portable user terminal 1, forexample.

In the upper portion of this virtual home viewer window 101, “Close”button 102, which is pressed to close the window, “Maximize” button 103,which is pressed to display the window in full size or return the sizethereof to its original size, and “Minimize” button 104, which ispressed to minimize the window, are arranged.

The virtual home viewer window 101 is divided into a main panel 105 inwhich the image of the virtual home is displayed, a control panel 106 inwhich the information about an icon selected by the user from amongthose displayed in the main panel 105, and a communication panel 107 inwhich chat, mail text, and news, for example, are displayed.

The following describes the virtual home shown in the main panel 105with reference to FIG. 6. In the room of the virtual home (hereaftersometimes referred to as a virtual room), a mascot 121 exists thatautonomously moves around in the room and that visits virtual homes ofother users as an agent for this user, for example.

The mascot as used herein has the capabilities of invoking theimpression that the user is visiting the currently displayed place,drawing off user's attention to the change in the state of the virtualhome viewer program, allowing the user to visually recognize thevisiting own virtual home by an other user, and carrying an icon whenvisiting the virtual home of an other user. Also, the mascot has thecapabilities of acting for the user's processing in some manner as withthe agent disclosed in Japanese Patent Laid-open No. 2002-44765.However, the mascot need not always have these capabilities of acting asan agent for the user's processing.

The virtual room is arranged with a door icon 122, which is clicked whenthe mascot 121 visits another user's home, a mailbox icon 123, which isclicked when mail addressed to that user is received, a telephone icon124, which is clicked when talking (actually chatting) with anothermember of the member group to which this user belongs, a PC icon 125,which is clicked to control the really existing PC 12 in user A's home,and a TV set icon 126, which is clicked to control the TV set 13 reallyexisting in user A's home.

Further, the virtual room is arranged with a sofa icon 127 and a tableicon 128, which are furniture, and a potted plant icon 129 and a trashbox icon 130, which are indoor ornaments and small objects (hereafterreferred to as items).

The icons, the PC icon 125 through the trash box icon 130, can bearranged at any places in the room. The items icons such as the pottedplant icon 129 may be taken out of the room and presented to otherusers.

In addition, the virtual room is arranged with a closet icon 131, whichis clicked to accommodate item icons such as the potted plant icon 129(the accommodated icons are hidden from the screen) or to bring outaccommodated icons into the room.

The user can select the designs of icons, the patterns of virtual roomwallpapers, and the materials of the floor or the like from amongprepared options.

The following describes a sequence of processing operations beingcarried out when the CPU 31 of the portable user terminal 1 executes thevirtual home viewer program 51 loaded in the RAM 33. In what follows,the subject of the action of the portable user terminal 1 is the virtualhome viewer program 51.

First, a login operation is performed on the equipment on the serviceprovider side. The following describes this login processing withreference to the flowchart shown in FIG. 7.

When the virtual home viewer program 51 is started up, the external viewof user A's home is displayed in the main panel 105 of the screen asshown in FIG. 8. For the displaying of this image, the layoutinformation supplied from the virtual home DB server 21 at the lastlogin and cached (or held) in the storage block 39 or the like is used.A password entry box 108 and “Enter Home” button 109 are displayed inthe control panel 106.

When user A enters the login password in the password entry box 108 andclicks the “Enter Home” button 109, the virtual home viewer program 51accesses the virtual home DB server 21 (shown in FIG. 1) via the basestation 4 and the Internet 5 to send a login request in step S1 (of FIG.7).

In response to this login request, the virtual home DB server 21 sendsthe URL (Uniform Resource Locator) of the authentication server 22 backto the virtual home viewer program 51 via the Internet 5 and the basestation 4 in step S11.

In step S2, the virtual home viewer program 51 accesses theauthentication server 22 via the base station 4 and the Internet 5 tosend the password and ID (the personal ID preset for user A or thedevice ID set to the portable user terminal 1) being entered by user Aalong with a verification request.

In response to this verification request, the authentication server 22checks the validity of the password and ID. If the password or ID arefound to be valid, the server 22 supplies the access right to thevirtual home viewer program 51 via the Internet 5 and the base station4, thereby allowing access to all devices on the service provider sidein step S21.

In step S3, the virtual home viewer program 51 sends the granted accessright to the virtual home DB server 21 via the base station 4 and theInternet 5. In step S12, the virtual home DB server 21 requests theauthentication server 22 for verifying the access right send from thevirtual home viewer program 51. In step S22, the authentication server22 verifies the access right and sends the result of the verification tothe virtual home DB server 21.

In step S13, if the verification result received from the authenticationserver 22 indicates that the access right is authentic, then the virtualhome DB server 21 notifies the virtual home viewer program 51 of thesuccessful login operation via the Internet 5 and the base station 4.

Receiving the successful login notification, the virtual home viewerprogram 51 can continue the subsequent processing. This is the end ofthe description of the login processing.

Next, the processing of getting the layout information for displayingthe virtual room in the main panel 105 is executed. The followingdescribes this layout information acquisition processing with referenceto the flowchart shown in FIG. 9.

In step S31, the virtual home viewer program 51 requests the virtualhome DB server 21 for the layout information about the user A's virtualhome via the base station 4 and the Internet 5. In response to thisrequest, the virtual home DB server 21 supplies the most recent layoutinformation about the user A's virtual home stored in the virtual homeDB server 21 to the virtual home viewer program 51 via the Internet 5and the base station 4 in step S41.

In step S32, the virtual home viewer program 51 compares the capturedmost recent layout information with the layout information cached in thestorage block 39 or the like and requests the virtual home DB server 21for the updated data such as icon image data, which do not exist in thecached layout information but exist in the most recent layoutinformation, via the base station 4 and the Internet 5.

In response to this request, the virtual home DB server 21 supplies therequested updated data such as icon image data to the virtual homeviewer program 51 via the Internet 5 and the base station 4 in step S42.Then, in step S33, the virtual home viewer program 51 displays the imageof the virtual room in the main panel 105 as shown in FIG. 10. At thismoment, because the user selects none of the icons in the virtual room,the state (age of home, total login time, wall type, floor type, etc.)of the virtual home is displayed in the control panel 106. This is theend of the description of the layout information acquisition processing.

Next, the processing of capturing the event information of user A isexecuted. The following describes user A's event information acquisitionprocessing with reference to the flowchart shown in FIG. 11. The eventinformation is stored in the virtual home DB server 21 (shown in FIG. 1)and indicates an action performed on user A by an other user (forexample, the arrival of mail to user A, the recording of answerphonemessage for user A, or the visiting by an other user to user A's virtualhome).

In step S51, the virtual home viewer program 51 requests the virtualhome DB server 21 for the event information of user A via the basestation 4 and the Internet 5. In response to this request, the virtualhome DB server 21 supplies the event information of user A via theInternet 5 and the base station 4 in step S61.

In step S52, on the basis of the user A's event information suppliedfrom the virtual home DB server 21, the virtual home viewer program 51accesses the user information holding server 24 via the base station 4and the Internet 5 to request for the event detail informationcorresponding to the event information. The event detail informationherein denotes the text data, which are specific contents of mail ormessage addressed to user A.

However, in this case, this is the first connection to the userinformation holding server 24 since the login, so that, like theprocessing of steps S3, S12, and S22 of the above-mentioned loginprocessing, such a sequence of processing operations is executed beforerequesting for the event detail information corresponding to the eventinformation as that the virtual home viewer program 51 sends the accessright obtained from the authentication server 22 to the user informationholding server 24, the user information holding server 24 requests theauthentication server 22 for the verification of the access right sentfrom the virtual home viewer program 51, and the authentication server22 verifies the access right and sends the result of the verification tothe user information holding server 24.

In step S71, the user information holding server 24 supplies therequested event detail information to the virtual home viewer program 51via the Internet 5 and the base station 4. This is the end of thedescription of the event information acquisition processing.

It should be noted that this event information acquisition processingwill be periodically executed hereafter in predetermined timing; namely,this processing is executed in a polling manner.

Next, home network information acquisition processing is executed tounderstand the state of the equipment components constituting the homenetwork 2 of user A. The following describes this home networkinformation acquisition processing with reference to the flowchart shownin FIG. 12.

In step S81, the virtual home viewer program 51 accesses the home server11 (shown in FIG. 1) via the base station 4 and the Internet 5 torequest for the searching of the home network 2. In response to thisrequest, the home server 11 searches for the home network 2 in step S91.To be more specific, the home server 11 requests all the devicesconnected thereto via the data bus 14 for a response.

In response to this request, the TV set 13 sends the information foridentifying itself to the home server 11 in step S101. It should benoted that the PC 12 connected to the home server 11 via the data bus 14executes the substantially the same processing, although not shown.Subsequently, the PC 12 executes the substantially the same processingas the TV set 13, the description of which is skipped.

In step S92, on the basis of the response from the TV set 13 and the PC12 connected to the data bus 14, the home server 11 creates a list ofhome network services indicative of a list of devices constituting thehome network 2 and supplies the created list to the virtual home viewerprogram 51 via the Internet 5 and the base station 4.

On the basis of the home network service list received from the homeserver 11, the virtual home viewer program 51 requests the home server11 via the base station 4 and the Internet 5 for the detail informationabout the devices (also referred to as network devices) constituting thehome network 2 in step S82. The detail information about the networkdevices herein includes the items that can be controlled from thevirtual home viewer program 51 (in the case of TV set 13, power on/off,volume up/down, channel change, multiplexed audio switching, and inputsignal switching, for example; in the case of the PC 12, the realizablevarious operations of the HDD video recorder 92 (shown in FIG. 4), theaudio data transmission service 93, and photograph data transmissionservice 94, for example) and the information about the operator buttonsbeing displayed in the control panel 106 when controlling networkdevices.

In response to this request, the home server 11 requests the TV set 13constituting the home network 2 for the detail information in step S93.

In response to this request, the TV set 13 supplies the detailinformation to the home server 11 in step S102. In step S94, the homeserver 11 supplies the detail information received from each networkdevice to the virtual home viewer program 51 via the Internet 5 and thebase station 4.

In step S83, the virtual home viewer program 51 requests the home server11 via the base station 4 and the Internet 5 for the current statusinformation of each network device. In response to this request, thehome server 11 requests the TV set 13 for the current status informationin step S95.

In step S103, in response to this request, the TV set 13 supplies thecurrent status information to the home server 11. The current statusinformation of the TV set 13 includes a channel being tuned in and avolume setting, for example. In step S96, the home server 11 suppliesthe current status information received from each network device to thevirtual home viewer program 51 via the Internet 5 and the base station4. This is the end of the description of the home network informationacquisition processing.

It should be noted that, on the basis of the current status informationreceived from each network device, the virtual home viewer program 51changes the displaying of icons in the main panel 105 representative ofthe network devices arranged in the virtual room. For example, if the TVset 13 of user A is receiving a television program, an image isdisplayed on the screen of the TV set icon 126.

Next, in order to obtain the member list of the user group to which userA belongs, member list acquisition processing is executed. The followingdescribes this member list acquisition processing with reference to theflowchart shown in FIG. 13.

In step S111, the virtual home viewer program 51 accesses theinter-user-terminal communication relay server 23 via the base station 4and the Internet 5 to send the server 23 the access right granted fromthe authentication server 22 in the above-mentioned login processing,thereby requesting for the member list.

In response to this request, in step S121, the inter-user-terminalcommunication relay server 23 requests the authentication server 22 forthe verification of the access right received from the virtual homeviewer program 51. In response to this request, the authenticationserver 22 verifies the access right and supplies the result of theverification to the user information holding server 24 in step S131.

In step S122, if the verification result received from theauthentication server 22 indicates that the access right is authentic,the inter-user-terminal communication relay server 23 supplies themember list of the user group to which user A belongs to the virtualhome viewer program 51 via the Internet 5 and the base station 4. Thisis the end of the description of the member list acquisition processing.

After the completion of the sequence of processing operations describedso far, various standalone processing and communication processing (eachwill be described later) are ready for execution.

The following describes logout processing for exiting the virtual homeviewer program 51 with reference to the flowchart shown in FIG. 14. Thelogout processing starts when the user clicks “Close” button 102 (asshown in FIG. 5) in the virtual home viewer window 101. Alternatively,the logout processing starts when “End” button 137, among variousbuttons (to be detailed later with reference to FIG. 75), which aredisplayed in the control panel 106 upon clicking of the door icon 122shown in the main panel 105, is clicked as shown in FIG. 15.

In step S141, the virtual home viewer program 51 sends the currentlayout information (indicative of the arrangement and orientation ofeach icon in the virtual room) to the virtual home DB server 21 via thebase station 4 and the Internet 5, thereby requesting for the updatingof the layout information stored in the virtual home DB server 21. Instep S151, in response to this update request, the virtual home DBserver 21 updates the layout information stored therein. Then, thevirtual home DB server 21 supplies the virtual home viewer program 51with the completion of the updating of the current layout informationvia the Internet 5 and the base station 4.

In step S142, the virtual home viewer program 51 sends the personal dataincluding the text data such as memo entered by use of the notepadcapability (to be described later with reference to FIGS. 47 through 50)to the user information holding server 24 via the base station 4 and theInternet 5 to request for the updating of the personal data stored inthe user information holding server 24. In step S161, in response tothis update request, the virtual home DB server 21 updates the personaldata of user A stored therein. Subsequently, the virtual home DB server21 notifies the virtual home viewer program 51 of the completion of theupdating of the personal information via the Internet 5 and the basestation 4.

In step S143, the virtual home viewer program 51 requests the virtualhome DB server 21 for logout via the base station 4 and the Internet 5.In step S152, in response to this logout request, the virtual home DBserver 21 sets itself so as to reject any access from the virtual homeviewer program 51 until logged in again and notifies the virtual homeviewer program 51 of the completion of logout via the Internet 5 and thebase station 4.

In step S144, the virtual home viewer program 51 requests the userinformation holding server 24 for logout via the base station 4 and theInternet 5. In step S162, in response to this logout request, the userinformation holding server 24 sets itself so as to reject any accessfrom the virtual home viewer program 51 until logged in again andnotifies the virtual home viewer program 51 of the completion of logoutvia the Internet 5 and the base station 4.

In step S145, the virtual home viewer program 51 requests theinter-user-terminal communication relay server 23 for logout via thebase station 4 and the Internet 5. In step S171, in response to thislogout request, the inter-user-terminal communication relay server 23sets itself so as to reject any access from the virtual home viewerprogram 51 until logged in again and notifies the virtual home viewerprogram 51 of the completion of logout via the Internet 5 and the basestation 4. In response to this notification of logout completion, theprocessing of the virtual home viewer program 51 is completed. This isthe end of the description of the login processing.

Next, standalone processing executable after a login sequence will bedescribed.

By dragging and dropping icons to a desired position in the virtualroom, the user can change the arrangement of desired icons (however,some icons such as the mailbox icon 123 (shown in FIG. 6), which belongsto the door icon 122, cannot be moved).

For example, the user can drag the sofa icon 127 of the virtual roomshown in FIG. 16 and drop the icon at the upper right side of the tableicon 128, thereby changing the arrangement of the sofa icon 127 as shownin FIG. 17.

In addition, by double-clicking a desired icon in the virtual room, theuser can switch between its orientations, the left-oblique and theright-oblique toward the user (however, some icons cannot be changed inorientation).

For example, double-clicking the sofa icon 127 in the right-obliqueorientation in the virtual room shown in FIG. 17 can switch the positionto the left-oblique orientation as shown in FIG. 18.

For example, again, dragging a board game set icon 141 in the virtualroom shown in FIG. 18 and dropping it on the table icon 128 can arrangethe board game set icon 141 on the table icon 128 as shown in FIG. 19.

It should be noted that the board game set icon 141 may be arrangedbehind the table icon 128. To be more specific, dragging the board gameset icon 141, from the lower right to the upper left in the screen, tothe table icon 128 and dropping the board game set icon 141 thereon canarrange the board game set icon 141 on the table icon 128 as shown inFIG. 20A.

Conversely, dragging the board game set icon 141, from the upper left tothe lower right in the screen, to the table icon 128 and dropping theboard game set icon 141 thereon can arrange the board game set icon 141behind the table icon 128 as shown in FIG. 20B.

Every time the arrangement or orientation of each icon has been changedas described above, the layout information is updated accordingly andthe updated layout information is sent to the virtual home DB server 21in a predetermined period.

The user can drag a desired icon in the virtual room and drop the icononto the closet icon 131, thereby accommodating the icon into the closet(however, some icons cannot be accommodated in the closet).

For example, as shown in FIG. 21, when the board game set icon 141 isdragged to the closet icon 131, the door of the closet icon 131 opens.When the board game set icon 141 is dropped in this state, the boardgame set icon 141 is accommodated in the closet.

When an icon has been accommodated in the closet icon 131, the closeticon 131 becomes the selected state (in the figure, the framed state) asshown in FIG. 22. When the closet icon 131 is in the selected state, apopup list 151 is displayed adjacent to the closet icon 131.

The popup list 151 shows the accommodated icons in reduced size in theorder of accommodation and up to the predetermined number (five in thecase of FIG. 22). If more than the predetermined number of icons areaccommodated in the closet, a scroll button 159 is shown in the popuplist 151. This scroll button 159 is clicked to display all theaccommodated icons by scrolling.

When the closet icon 131 is in the selected state, the control panel 106shows “Display Item List” button 157 and “Close” button 158.

The “Display Item List” button 157 is clicked when displaying all iconsaccommodated in the closet into the main panel 105 as shown in FIG. 23.The “Close” button 158 is clicked when returning the display of the mainpanel 105 to the virtual room when the item list is displayed in themain panel 105.

As shown in FIG. 24, when the user selects one of the items in the itemlist shown in the main panel 105 (in the example of FIG. 24, the boardgame set icon 141 accommodated in the closet), the control panel 106shows the property of the selected icon and “Close” button 161 which isclicked to return the display of the main panel 105 to the virtual room.

However, in the example of FIG. 24, the property of the selected boardgame set icon 141 does not especially exist, so that only the “Close”button 161 is shown. When the “Close” button 161 is clicked, the displayof the main panel 105 is returned to the virtual room as shown in FIG.25.

By dragging a desired icon from the popup list 151 and dropping the iconat a desired place in the virtual room, the user can arrange any iconsaccommodated in the closet into the virtual room again. For example,FIG. 26 shows that an ashtray icon 154 accommodated in the closet isarranged on the table icon 128 in the virtual room.

Referring to FIG. 27, there is shown a state in which the PC icon 125 isselected in the virtual room shown in the main panel 105. When the userselects the PC icon 125, the control panel 106 shows a “Video Recorder”button 161, “Audio Player” button 162, and “Photo Album” button 163. Thebutton correspond to the capabilities realized by the PC 12 in user A'shome controllable from the virtual home viewer program 51.

The “Video Recorder” button 161 is clicked when controlling the HDDvideo recorder 92, which can be realized by the PC 12. The “AudioPlayer” button 162 is clicked when controlling the audio datatransmission service 93, which can be realized by the PC 12. The “PhotoAlbum” button 163 is clicked when controlling the photograph datatransmission service 94, which can be realized by the PC 12.

In addition to the PC icon 125 corresponding to the PC 12, the controlpanel 106 shows “Divide Function” button 164 that is clicked to show theicons, which correspond to the HDD video recorder 92, the audio datatransmission service 93, and the photograph data transmission service94, in the virtual room. Namely, when the “Divide Function” button 164is clicked, a video recorder icon 171, an audio player icon 172, and aphoto album icon 173 appear in an empty place on the floor in additionto the PC icon 125. As with the other icons, the user can change thearrangement of the video recorder icon 171, the audio player icon 172,and the photo album icon 173 as desired.

Referring to FIG. 29, there is shown a state in which the arrangement ofthe video recorder icon 171 and the audio player icon 172 has beenchanged from the state shown in FIG. 28 and the video recorder icon 171is selected by the user. When the video recorder icon 171 is selected,the control panel 106 shows the current state of the HDD video recorder92 as the property of the video recorder icon 171. It should be notedthat, in order to show the current state of the HDD video recorder 92,the current status information of the PC 12 obtained by the home networkinformation acquisition processing executed earlier is used.

In addition, the control panel 106 shows “TV Program Guide” button 181being clicked to show a television program guide (hereafter referred toas a TV program guide) in the main panel 105, “Check Preset Recording”button being clicked to show the information about preset recording inthe main panel 105, “Recorded Program List” button 183 being clicked toshow a list of recorded programs in the main panel 105, and “Close”button 184 being clicked to return the display of the main panel 105 tothe display of the virtual room.

Referring to FIG. 30, there is shown a state in which a TV program guideis shown in the main panel 105 in response to the clicking of the “TVProgram Guide” button 181. For the displaying of a TV program guide, theEPG information obtained by the virtual home viewer program 51 from theEPG server 25 is used.

The following describes the EPG information acquisition processing beingexecuted by the virtual home viewer program 51, with reference to theflowchart shown in FIG. 31. In step S181, the virtual home viewerprogram 51 accesses the EPG server 25 via the base station 4 and theInternet 5 to request for EPG information. In response to this request,the EPG server 25 supplies the requested EPG information to the virtualhome viewer program 51 via the Internet 5 and the base station 4 in stepS191. This is the EPG information acquisition processing.

As shown in FIG. 32, when one of the program names listed in the TVprogram guide shown in the main panel 105 is selected by the user, thecontrol panel 106 shows “Record Mode” setting box 191 being clicked toset the recording mode for recording the selected program to thestandard mode, the high quality mode, or the long time mode, “Record”button 192 being clicked to preset the recording of the selectedprogram, and “Close” button 193 being clicked to return the display ofthe main panel 105 to the state in which the video recorder icon 171 isselected in the virtual room.

As shown in FIG. 33, when the “Record” button 192 is clicked with one ofthe program names listed in the TV program guide shown in the main panel105 selected, the virtual home viewer program 51 sends to the homeserver 11, via the base station 4 and the Internet 5, a control signalfor instructing the HDD video recorder 92 to preset the recording of theselected program. The home server 11 sends this control signal to the PC12 via the data bus 14. Receiving this control signal, the HDD videorecorder 92 of the PC 12 executes the preset recording in accordancewith this control signal. Thus, user A can remotely control the presetrecording of TV programs through the PC 12 of user A's home from theoperator screen of the virtual home viewer program 51, which is executedon the portable user terminal 1.

Returning to the description of the control panel 106 when the videorecorder icon 171 is selected. Clicking “Recorded Program List” button183 causes the main panel 105 to show a list of programs recorded by theHDD video recorder 92 of the PC 12.

It should be noted that the processing by the virtual home viewerprogram 51 executed on the portable user terminal 1 can also be executedby the virtual home viewer 91, which is realized by executing thevirtual home viewer program 81 by the PC 12 of user A's home.

The following describes the processing being executed by the virtualhome viewer 91, associated with the video recorder icon 171, and isconclusive only in user A's home.

As shown in FIG. 34, when the “Recorded Program List” button 183 isclicked in the state in which the virtual room is displayed in the mainpanel 105 of the virtual home viewer 91, the video recorder icon 171 isselected, and the “TV Program Guide” button 181 through the “Close”button 184 are displayed in the control panel 106, a recorded programlist indicative of the programs already recorded by the HDD videorecorder 92 is shown in the main panel 105 of the virtual home viewer 91as shown in FIG. 35.

In order to display this recorded program list, the virtual home viewer91 gets content list information from the HDD video recorder 92. Thefollowing describes this content list information acquisition processingwith reference to the flowchart shown in FIG. 36.

In step S201, the virtual home viewer 91 requests the HDD video recorder92 for content list information. In step S211, in response to thisrequest, the HDD video recorder 92 supplies the content list informationto the virtual home viewer 91. This is the end of the description of thecontent list information acquisition processing.

If the user selects one of the program names in the recorded programlist shown in the main panel 105, the control panel 106 shows, as shownin FIG. 37, the information about the contents of the selected recordedprogram, “Reproduce” button 194 being clicked to reproduce the selectedrecorded program, “Pause” button 195 being clicked to temporarily stopreproduction, “Stop” button 196 being clicked to stop reproduction,“Delete” button 197 being clicked to delete the selected recordedprogram, and “Close” button 198 being clicked to return the display ofthe main panel 105 to the state in which the video recorder icon 171 isselected in the virtual room.

As shown in FIG. 37, when the “Reproduce” button 194 is clicked with oneof the recorded programs in the recorded program list shown in the mainpanel 105 selected, “Main Panel” button 194A and “TV Set” button 194Bfor switching between the destinations to which the TV program to bereproduced is outputted are shown adjacent to the “Reproduce” button194.

When “Main Panel” button 194A is clicked, the video of the TV program tobe reproduced is displayed in the main panel 105 as shown in FIG. 39.The following describes this processing of displaying the reproducedvideo of each recorded program in the main panel 105, with reference tothe flowchart shown in FIG. 40.

In step S221, the virtual home viewer 91 requests the HDD video recorder92 for the stream reproduction of a selected recorded program. In stepS231, in response to this request, the HDD video recorder 92 startsreproducing the recorded program and supplies its stream data to thevirtual home viewer 91.

Next, when the user clicks “Pause” button 195 or the “Stop” button 196shown in the control panel 106 of the virtual home viewer 91, thevirtual home viewer 91 requests the HDD video recorder 92 to stopreproduction in step S222. In step S232, in response to this request,the HDD video recorder 92 stops reproduction of the recorded program,thereby stopping the supply of the stream data to the virtual homeviewer 91 and notifies the virtual home viewer 91 of the stop ofsupplying the stream data. This is the end of the description of theprocessing of displaying a recorded program onto the main panel.

Now, returning to FIG. 38, if “TV Set” button 194B is clicked, characterstring “Recorded Program List” so far displayed in the upper portion ofthe main panel 105 is superseded with character string “Now showing onTV” as shown in FIG. 41. When the “Close” button 198 is clicked in thisstate, the display of the main panel 105 is returned to the virtual roomas shown in FIG. 42 and the video of the reproduced recorded program isdisplayed on the screen of the TV set icon 126.

If really existing “TV Set” button 194B is clicked, the video of therecorded program reproduced by the HDD video recorder 92 is displayed onthe TV set 13.

The following describes the processing of displaying the video of arecorded program reproduced by the HDD video recorder 92 onto the TV set13, with reference to the flowchart shown in FIG. 43.

In step S241, the virtual home viewer 91 outputs a control signal to theTV set 13 via the data bus 14 that instructs the TV set 13 to get thestream data of the recorded program from the HDD video recorder 92. Instep S251, in response to this control signal, the TV set 13 requeststhe HDD video recorder 92 via the data bus 14 for the streamreproduction of the recorded program. In step S261, the HDD videorecorder 92 notifies the TV set 13 via the data bus 14 of theacknowledgement for this request. In step S252, the TV set 13 notifiesthe virtual home viewer 91 via the data bus 14 of the acknowledgement bythe HDD video recorder 92 for the stream reproduction.

In step S262, the HDD video recorder 92 starts reproducing the recordedprogram and supplies the stream data to the TV set 13 via the data bus14.

Next, when the user clicks the “Pause” button 195 or the “Stop” button196 of the control panel 106 of the virtual home viewer 91, the virtualhome viewer 91 outputs, in step S242, a control signal to the TV set 13via the data bus 14 that instructs the TV set 13 to stop the acquisitionof the stream data of the recorded program from the HDD video recorder92. In step S252, in response to this control signal, the TV set 13requests the HDD video recorder 92 via the data bus 14 for stopping thesupply of the stream data.

In step S263, in response to this request, the HDD video recorder 92stops reproducing the recorded program, thereby stopping the supply ofthe stream data to the TV set 13, notifying the TV set 13 of thestopping of the supply of the stream data. In step S253, TV set 13notifies the virtual home viewer 91 via the data bus 14 of theacknowledgement by the HDD video recorder 92 of the stopping of thestream data supply. This is the end of the description of theabove-mentioned processing.

Referring to FIG. 44, there is shown a state in which the user selectsTV set icon 126 in the virtual room in the main panel 105. Selection ofthe TV set icon 126 displays, on the control panel 106, the state of theTV set 13, which really exists in user A's home (currently, nothing isdisplayed because the power to the TV set 13 is off), and the buttonsfor controlling the TV set 13, namely, “POWER” button 211 for tuningon/off the power, numeric buttons 212 for specifying channels of the TVset 13, and “Volume” button 213 for controlling the volume as theproperty of the TV set icon 126.

When the user clicks the “POWER” button 211, the virtual home viewer 91outputs a control signal to the home server 11 to turn on the power tothe TV set 13. The home server 11 relays this control signal to the TVset 13. In response to this control signal from the virtual home viewer91, the TV set 13 turns on its power, starting the reception of thechannel (for example, 10 ch) currently set.

In response to the TV set 13, the virtual home viewer 91 changes the TVset icon 126 to the power-on state as shown in FIG. 45. Namely, an imageis displayed on the screen of the TV set icon 126. Also, the virtualhome viewer 91 displays the channel currently received (10 ch in thisexample) and the information associated with that program onto thecontrol panel 106 as the property of the TV set icon 126.

Next, when the user clicks numeric key 212-6 to specify the reception of6 ch, for example, the virtual home viewer 91 outputs a control signalto the home server 11 to instruct the TV set 13 to receive 6 ch. Thehome server 11 relays this control signal to the TV set 13. The TV set13 responds to the virtual home viewer 91 and switches channel from 10ch to 6 ch.

In response to the TV set 13, the virtual home viewer 91 changes theimage shown on the screen of the TV set icon 126 as shown in FIG. 46.Also, the virtual home viewer 91 displays on the control panel 106 thechannel currently received by the TV set 13 (6 ch in this example) andthe information about that program as the property of the TV set icon126.

The following describes a simplified memo capability using a notepadicon 231 (FIG. 47) of the virtual room. Referring to FIG. 47, there isshown a state in which the notepad icon 231 on the table icon 128 isselected in the virtual room shown in the main panel 105. When thenotepad icon 231 is selected, the control panel 106 displays the headportion “ 4/13Baseball Training . . . ” of the page (in this example,page 3/500) opened last, as the property of the notepad icon 231.

The control panel 106 also displays “Page” button 232 being clicked toselect a page of the notepad to be displayed as the property, “New”button 233 being clicked to open a new page in the main panel 105,“Open” button 234 being clicked to display the entire selected page asthe property in the main panel 105, “Cut” button 235 being clicked tocut a selected page from the notepad to generate a separate memo sheeticon 241 (FIG. 50), and “Close” button 236 being clicked to return fromthe opened page of the notepad to the virtual room.

Referring to FIG. 48, there is shown a state in which a new page isdisplayed in the main panel 105 upon clicking of the “New” button 233.The user can enter text into the page displayed in the main panel 105 asshown in FIG. 49.

Referring to FIG. 50, there is shown a state in which, upon clicking ofthe “Cut” button 235, a selected page is cut from the notepad and thememo sheet icon 241 corresponding to one sheet of notepad has appearedseparate from the notepad icon 231. At this moment, in the virtual roomshown in the main panel 105, the memo sheet icon 241 is selected and thecontrol panel 106 displays the head portion (“This is a test of notepad. . . ”) of the text written in the memo sheet 241 as the property ofthe memo sheet icon 241. In addition, the control panel 106 displays“Open” button 242 being clicked to display the text written on thenotepad onto the main panel 105 and “Close” button 243 being clicked toreturn the display of the main panel 105 from the text to the virtualroom.

The memo sheet icon 241 may be passed to other users in thecommunication processing, which will be described later.

It should be noted that the text written in each page of the notepad,the arrangement and orientation of the newly created memo sheet icon241, and the text written on the memo sheet are uploaded to the userinformation holding server 24 at predetermined time intervals as theuser's personal data.

This is the end of the description of various standalone processingoperations, not associated with other users.

The following describes various communication processing operationsbased on the virtual space remote control system in which a plurality ofusers involve. The parties with whom user A can execute communicationprocessing are restricted to the members of the user group to which userA belongs.

By selecting the telephone icon 124 in the virtual room and clickingbuttons shown in the control panel 106, user A can talk (actually,exchange messages by chat) with any member of the user group to whichuser A belongs.

When the telephone icon 124 in the virtual room is selected as shown inFIG. 51, the control panel 106 displays parties whom user A can call,namely, a called party list 251 indicative of the names of members inthe user group to which user A belongs, as the property of the telephoneicon 124.

The called party list 251 shows the names of members of the user groupto which user A belongs, in the predetermined number. If the number ofmembers of this user group exceeds a predetermined number, a scrollbutton (not shown) is displayed by which all member names are scrolledto be shown. Beside each member name, a call button is shown. Forexample, when the call button for “Yumiko” is clicked, user A can talkwith “Yumiko” who is a member of the same user group as user A (detailsof which will be described later).

It should be noted that, for the displaying of the called party list251, the member list obtained in the startup sequence is referenced.When the called party list 251 is displayed, the member list acquisitionprocessing described above with reference to FIG. 13 is executed againto get the most recent member list, thereby updating the display of thecalled party list 251.

Also, the control panel 106 displays “Member List” button 252 beingclicked to display on the main panel 105 the list of members of the usergroup to which user A belongs, “Disconnect” button 253 being clicked toend talk, “Listen Recorded Message” button 254 being clicked to listento a recorded message (actually, text data) recorded by a user whocalled user A, and “Close” button 255 being clicked to return thedisplay of the main panel 105 from the contents of recorded message tothe virtual room.

The following describes the processing of adding new members to the usergroup to which user A belongs. When the “Member List” button 252 isclicked, the main panel 105 displays a list of members of the user groupto which user A belongs as shown in FIG. 52. The member list shown inthe main panel 105 includes “User Search” button 261 in addition tomember names.

When the “User Search” button 261 is clicked, the control panel 106 isswitched from the display of the called party list 251, for example, tothe display for user search, as shown in FIG. 53. To be more specific,the control panel 106 displays an entry box 271 in which a user name isentered as a search condition, “Search” button 272 being clicked tostart a search operation and display a search result in the main panel105, “Member List” button 271 being clicked to return the display of themain panel 105 from search result to member list, and “Close” button 274being clicked to return the display of the main panel 105 to the virtualroom.

For example, as shown in FIG. 54, when “Nanako” is entered in the entrybox 271 and the “Search” button 272 is clicked, search condition“Nanako” is sent to the inter-user-terminal communication relay server23 via the base station 4 and the Internet 5. The inter-user-terminalcommunication relay server 23 executes a search operation on all usersand sends back the search result, thereby displaying a list of usernames including “Nanako” in the main panel 105.

If user name “Nanako” displayed as the search result in the main panel105 is selected by the user as shown in FIG. 55, for example, thecontrol panel 106 displays the property of the selected user name“Nanako” (the text prepared and intended for public display by “Nanako”herself), “Register Member” button 281 being clicked to additionallyregister the user having selected user name with the user group, and“Close” button 282 being clicked to return the display of the main panel105 to the member list.

When “Register Member” button 281 is clicked with user name “Nanako”selected in the search result shown in the main panel 105, a request forprovisionally registering “Nanako” with the user group to which user Abelongs is sent to the inter-user-terminal communication relay server 23via the base station 4 and the Internet 5. After the provisionalregistration of “Nanako” by the inter-user-terminal communication relayserver 23, the display of the main panel 105 is switched to the memberlist, thereby additionally displaying user name “Nanako (Provisional)following existing member names. It should be noted that the details ofthis processing will be described later with reference to FIG. 60.

The following describes the processing of deleting a member from theuser group to which user A belongs. For example, as shown in FIG. 57, if“Hideo” is selected from the members of the user group with the memberlist displayed in the main panel 105, the control panel 106 displays theproperty (the text prepared and intended for public display by “Hideo”himself), “Delete Member” button 291 being clicked to delete the userhaving the selected user name from the user group and “Close” button 292being clicked to return the display of the main panel 105 to the virtualroom.

When the “Delete Member” button 291 is clicked with “Hideo” selected inthe member list displayed in the main panel 105, a request for deleting“Hideo” from the user group to which user A belongs is sent to theinter-user-terminal communication relay server 23 via the base station 4and the Internet 5. In response, the inter-user-terminal communicationrelay server 23 deletes “Hideo” from the user group and then “Hideo” isdeleted from the member list shown in the main panel 105 as shown inFIG. 58.

The following describes the processing being executed when “Nanako” hasbeen provisionally registered by user A with the user group to whichuser A belongs (in what follows, “Nanako” will be referred to as user Buntil otherwise noted). Before describing this processing, the virtualroom of user B will be described with reference to FIG. 59.

FIG. 59 shows the window of a virtual home viewer being displayed on thescreen of the PC 3, which is operated by user B. This window is alsodivided into a main panel 301, a control panel 302, and a communicationpanel 303. The main panel 301 shows the virtual room of user B. Like thevirtual room of user A, the virtual room of user B is arranged with amascot 321, a door icon 322, a mailbox icon 323, a telephone icon 324,and a PC icon 325, for example.

The following describes the above-mentioned processing being executedwhen user B has been provisionally registered by user A with the usergroup to which user A belongs, with reference to the flowchart shown inFIG. 60. In step S271, the virtual home viewer program 51 (hereafterreferred to as virtual home viewer A) of the portable user terminal 1 ofuser A requests the inter-user-terminal communication relay server 23for the provisional registration of user B via the base station 4 andInternet 5. In response to this request, in step S281, theinter-user-terminal communication relay server 23 provisionallyregisters user B with the user group to which user A belongs andnotifies the virtual home viewer A of the completion of the provisionalregistration.

The user B recognizes its provisional registration with the user groupto which user A belongs only when user B obtains the member list of theuser group to which user B belongs from the inter-user-terminalcommunication relay server 23.

To be more specific, in step S291, the virtual home viewer program ofthe PC 3 of user B (hereafter referred to as a virtual home viewer B)requests the inter-user-terminal communication relay server 23 for themember list via the Internet 5 as a startup sequence. In response tothis request, in step S282, the inter-user-terminal communication relayserver 23 sends, to virtual home viewer B via the Internet 5, the memberlist of the user group to which user B belongs (user A is not yetincluded in this member list) and a member registration permissionenquiry for confirming whether user B can be registered with the usergroup to which user A belongs.

When the member registration permission enquiry comes from theinter-user-terminal communication relay server 23, a member registrationform icon 326 is delivered to the mailbox icon 323 of the virtual roomof user B as shown in FIG. 59. When the user B chooses the memberregistration form icon 326, the control panel 302 displays theinformation indicative of the provisional registration of user B withthe user group to which user A belongs, “Permit Member Registration”button 311 being operated to permit the full registration and “RejectMember Registration” button 312 being operated to reject theregistration.

If user B clicks “Permit Member Registration” button 311 or “RejectMember Registration” button 312, then the virtual home viewer B notifiesthe inter-user-terminal communication relay server 23 via the Internet 5of the operation done by user B in step S292.

In response to this notification, in step S283, the inter-user-terminalcommunication relay server 23 fully registers user B with the user groupto which user A belongs or deletes user B from the user group of user Awith which user B is provisionally registered, notifying the virtualhome viewer B thereof via the Internet 5.

It should be noted that the notification of the full registration ofuser B is not explicitly sent to virtual home viewer A. However,subsequently, user A can determine whether user B has been fullyregistered as the member by checking the member list for the name ofuser B when the member list is displayed in virtual home viewer A.

This is the end of the description of the processing to be executed uponthe provisional registration of user B by user A with the user group towhich user A belongs.

The following describes the processing of talking (actually, chatting intext) with other users. As described above, talk can be executed betweenthe members of the same user group.

For example, to start talking with “Yumiko” who is the member of theuser group to which user A belongs, user A may only select the telephoneicon 124 in the virtual room to display the called party list 251 in thecontrol panel 106 as shown in FIG. 61 and click “Yumiko” button in themembers included in the called party list 251 as shown in FIG. 62.

When “Yumiko” button is clicked, the information about the currentstatus of “Yumiko” (hereafter referred to as user B until otherwisenoted) is obtained from the virtual home DB server 21. The followingdescribes this current status information acquisition processing withreference to the flowchart shown in FIG. 63. In step S301, virtual homeviewer A requests the virtual home DB server 21 via the base station 4and the Internet 5 for the current status of user B.

In response to this request, in step S311, the inter-user-terminalcommunication relay server 23 sends, to virtual home viewer A via theInternet 5 and the base station 4, the current status of user B; namely,user B is in the logout state (talk disabled), in the login state and athome (talk enabled), in the login state and at home but talking with another user (talk disabled), or in the login state and visiting an otheruser's virtual room (talk disabled). This is the end of the descriptionof the current status information acquisition processing.

If the current status information acquisition processing indicates thatuser B is in the login state and at home in the virtual room (talkenabled), virtual home viewer A starts telephone chat processing. Thefollowing describes this telephone chat processing with reference to theflowchart shown in FIG. 64.

In step S321, virtual home viewer A requests the virtual home DB server21 via the base station 4 and the Internet 5 for a telephone chat callevent for user B. In step S331, the virtual home DB server 21 generatesthe telephone chat call event for user B and notifies virtual homeviewer A via the Internet 5 and the base station 4 of theacknowledgement of this event request. In response to this notification,virtual home viewer A subsequently periodically requests theinter-user-terminal communication relay server 23 via the base station 4and Internet 5 for a telephone chat text, which is sent from user B touser A. It should be noted that, if user B does not answer the telephonechat call, nothing is sent back.

On the other hand, in step S341, by the event information acquisitionprocessing, which is periodically executed, virtual home viewer Brequests the virtual home DB server 21 via the Internet 5 for eventinformation. In response to this request, in step S332, the virtual homeDB server 21 notifies virtual home viewer B via the Internet 5 of thegeneration of a telephone chat event upon request by user A.

In response to this notification that a telephone chat event has beengenerated upon request by user A, virtual home viewer B displays thetelephone icon 324 in the virtual room shown in the main panel 301 in ahighlighted manner (the icon is partially lighted or entirely blinked)and generates the sound of the beep for the telephone as shown in FIG.65. Also, virtual home viewer B displays the name of telephone chatoriginator and “Answer” button 341 being clicked to answer the call oftelephone chat in the control panel 302.

If user B clicks the “Answer” button 341, the communication panel 303displays an text entry area 342 being entered the telephone chat text,“SEND” button 343 being clicked to send the text entered in the textentry area 342, and a text display area 344 being displayed thetelephone chat text.

When user B clicks the “Answer” button 341, enters text “Hello” in thetext entry area 342, and clicks the “SEND” button 343, then, in stepS342, virtual home viewer B sends the information that the telephonechat call has been answered and telephone chat text “Hello” to theinter-user-terminal communication relay server 23 via the Internet 5.

In response to this notification, in step S351, the inter-user-terminalcommunication relay server 23 returns the telephone chat text addressedto user B from user A. It should be noted that, in the phase of stepS351, the telephone chat text from user A to user B does not exist, sothat nothing is sent back.

On the other hand, virtual home viewer A waiting for the answer fromuser B to which telephone chat call was sent has been periodicallyrequesting the inter-user-terminal communication relay server 23 via thebase station 4 and the Internet 5 for telephone chat text to be sentfrom user B to user A. However, nothing has been sent back to virtualhome viewer B.

However, if virtual home viewer A requests the inter-user-terminalcommunication relay server 23 via the base station 4 and the Internet 5for telephone chat text to be sent from user B to user A in step S322,the inter-user-terminal communication relay server 23 sends, via theInternet 5 and the base station 4, information that user B has answeredthe telephone chat call and text “Hello” of the telephone chat sent fromuser B to user A to virtual home viewer A in response to this request instep S352.

In response to this notification, virtual home viewer A displays thattalk is being made with “Yumiko” in the called party list 251 of thecontrol panel 106 as shown in FIG. 66. Also, virtual home viewer Achanges the display of the communication panel 107 to a text entry area351, which is entered the telephone chat text, “SEND” button 352, whichis clicked to send the telephone chat text entered in the text entryarea 351, and a text display area 353, which is displayed the telephonechat text.

Subsequently, every time “SEND” button 352 is clicked, virtual homeviewer A displays, in the text display area 353, the telephone chat textaddressed to B entered in the text entry area 351, sends the telephonechat text to the inter-user-terminal communication relay server 23 viathe base station 4 and the Internet 5, obtains a telephone chat textaddressed to user A held in the inter-user-terminal communication relayserver 23 at that moment, and also displays the obtained telephone chattext in the text display area 353 as shown in FIG. 67. Further, virtualhome viewer A periodically gets the telephone chat text addressed touser A from the inter-user-terminal communication relay server 23 viathe base station 4 and the Internet 5 and also displays the obtainedtelephone chat text in the text display area 353. Virtual home viewer Balso executes the similar processing.

These processing operations by virtual home viewers A and B are executeduntil “Disconnect” button 253 shown in the control panel 106 is clickedby user A as shown in FIG. 68.

The following describes the telephone chat end processing to be executedwhen the “Disconnect” button 253 is clicked, with reference to theflowchart shown in FIG. 69. In step S361, virtual home viewer A notifiesthe inter-user-terminal communication relay server 23 via the basestation 4 and the Internet 5 of the disconnection, namely the end oftelephone chat.

In response to this notification, in step 371, the inter-user-terminalcommunication relay server 23 sends back the acknowledgement of thedisconnect operation and the telephone chat addressed to user A receivedup to this moment. In response, virtual home viewer A returns thedisplays of the main panel 105 and the control panel 106 to the state asit was before the starting of the telephone chat.

On the other hand, when virtual home viewer B periodically requests theinter-user-terminal communication relay server 23 via the Internet 5 fora telephone chat text addressed to user B in step S381, then, inresponse to this request, the inter-user-terminal communication relayserver 23 notifies virtual home viewer B via the Internet 5 of thedisconnection with user A, namely the end of the telephone chat in stepS372. In response to this notification, virtual home viewer B returnsthe displays of the control panel 302 and the communication panel 303 tothe state at it was before the starting of the telephone chat. This isthe end of the description of the telephone chat end processing.

The following describes answerphone message recording processing, whichis executed when user A leaves a message to user B because user B is inthe talk disabled state, with reference to FIG. 70. This answerphonerecording processing is executed if the current status of user Bobtained by the above-mentioned current status information acquisitionprocessing indicates that user B is in the login state and not at home(talk enabled) in the virtual home.

In step S391, virtual home viewer A displays (Answerphone) in the calledparty list 251 of the control panel 106 as the information indicativethat the called party is in the talk disabled state as shown in FIG. 71.Also, virtual home viewer A switches the display of the communicationpanel 107 to the display of talk chat, thereby displaying preset message“I'm away from home right now.” in the text display area 353.

At this moment, when user A enters a message (for example, “It's toobad, I wanted to visit you. I'll call again”) in the text entry area 351and clicks the “SEND” button 352, virtual home viewer A requests theuser information holding server 24 via the base station 4 and theInternet 5 for the storage of an object (the message to the answerphoneof user B) and sends the above-mentioned message “It's too bad, I wanted. . . ”

In response to this request, in step S401, the user information holdingserver 24 stores the message and notifies virtual home viewer A via theInternet 5 and the base station 4 of the acknowledgement of the requestfor storing the object.

In step S392, virtual home viewer A notifies the virtual home DB server21 via the base station 4 and the Internet 5 of the event information(the leaving of the message at the answerphone of user B). In responseto this notification, in step S411, the virtual home DB server 21notifies the virtual home viewer A via the Internet 5 and the basestation 4 of the reception of the event information.

Subsequently, if virtual home viewer B requests the virtual home DBserver 21 for event information as a periodical operation in step S421,then, the virtual home DB server 21 sends the event information (thatthere is a message left at the answerphone of user B) to virtual homeviewer B via the Internet 5 in step S412. It should be noted that thisevent information is held in the virtual home DB server 21 until virtualhome viewer B requests its deletion.

In response to this notification, in step S422, virtual home viewer Brequests the user information holding server 24 via the Internet 5 forthe message left at the answerphone of user B. In response to thisrequest, in step S402, the user information holding server 24 sends themessage left at the answerphone of user B to virtual home viewer B viathe Internet 5. It should be noted that, subsequently, the message leftat the answerphone of user B is held in the user information holdingserver 24 until virtual home viewer B requests its deletion.

In response, virtual home viewer B draws user B's attention to theexistence of the message left at its answerphone by blinking, forexample, the telephone icon 324 in the virtual room in the main panel301. When user B selects the blinking telephone icon 324, the controlpanel 302 displays a called party list 361 indicative of members of theuser group to which user B belongs, “Member List” button 362,“Disconnect” button 363, “Listen Recorded Message” button 364, and“Close” button 365 as shown in FIG. 72. It should be noted that thebuttons “Member List” through “Close” are similar to those buttons“Member List” 252 through “Close” 255 shown in FIG. 51, so that theirdescription will be skipped.

When the “Listen Recorded Message” button 364 in the control panel 302is clicked, the main panel 301 displays a recorded message list formedby boxes in each of which the name of the user who left a message andthe recording date thereof as shown in FIG. 73 (in the example shown inFIG. 73, only one box is used).

When one of the boxes of the recorded message list displayed in the mainpanel 301 is selected, the communication panel 303 displayscorresponding message “It's too bad, I wanted to visit you. I'll callagain.” as shown in FIG. 74. The control panel 302 displays “Delete”button 371 to be clicked to delete the record associated with themessage corresponding to the box selected in the recorded message listfrom the virtual home DB server 21 and the user information holdingserver 24 and “Close” button 372 to be clicked to return the display ofthe main panel 301 from the recorded message list to the virtual room.

When user B clicks the “Delete” button 371, virtual home viewer Brequests the virtual home DB server 21 via the Internet 5 for thedeletion of the event information (that there is a message left at theanswerphone of user B) in step S423. In response to this request, thevirtual home DB server 21 deletes the specified event information fromown record and then notifies virtual home viewer B via the Internet 5 ofthe completion of the requested deletion in step S413.

In step S424, virtual home viewer B requests the user informationholding server 24 via the Internet 5 for the deletion of thecorresponding message. In response to this message, in step S403, theuser information holding server 24 deletes the specified message fromown record and then notifies virtual home viewer B via the Internet 5 ofthe completion of the requested deletion. This is the end of thedescription of the answerphone recording processing.

The following describes the process of visiting the virtual room of amember by an other who belongs to the same user group, thereby chattingeach other and passing presents, for example, (it should be noted that,on the screen, the visit is made by the mascot). In the followingexample, user A visits the virtual room of user B.

When visiting the virtual room of an other member, user A may onlyselect the door icon 122 of the virtual room shown in the main panel 105and then click the button corresponding to the virtual home that user Awants to visit, among the buttons shown in response to the selection ofthe door icon 122 as its property, as shown in FIG. 75.

The following describes the buttons being displayed in the control panel106 as the property of the door icon 122. “Yumiko's Home” button 132 isclicked to make the mascot 121 visit the virtual room of “Yumiko.” Thesame holds with “Tsuyoshi's Home” button 133 and “Hideo's Home” button134. “Outdoor” button 135 is clicked to make the mascot 121 get out ofthe virtual room. “Lock” button 136 is clicked to lock the door of thevirtual room. “Logout” button 137 is clicked to end virtual home viewerA.

For example, the following describes conference generation processingbeing started upon clicking of “Yumiko's Home” button 132 as shown inFIG. 76, with reference to the flowchart shown in FIG. 77. In whatfollows, “Yumiko” belonging to the user group to which user A belongs isreferred to as user B until otherwise noted.

In step S431, virtual home viewer A notifies the virtual home DB server21 via the base station 4 and the Internet 5 of an event for requesting“generation of a conference hosted by user B.” A conference hereindenotes a collection of two or more users who share a virtual room.Various data are communicated as conference synchronous informationalmost realtime between the users belonging to the conference.

In response to the notification from virtual home viewer A, in stepS441, the virtual home DB server 21 stores the notified event andnotifies virtual home viewer A via the Internet 5 and the base station 4of the reception of the notification. Subsequently, virtual home viewerA periodically requests the inter-user-terminal communication relayserver 23 via the base station 4 and Internet 5 to get chat addressed touser A.

When virtual home viewer B requests the virtual home DB server 21 forevent information via the Internet 5 as a periodical operation in stepS451, then the virtual home DB server 21 sends the event information(the request for generating a conference hosted by user B) to virtualhome viewer B via the Internet 5 in step S442.

In response to the notification of this event information, in step S452,virtual home viewer B requests the inter-user-terminal communicationrelay server 23 via the Internet 5 for the generation of conference (B)to be hosted by virtual home viewer B ((B) denotes the term foridentifying a conference). In step S461, the inter-user—terminalcommunication relay server 23 generates conference (B) and notifiesvirtual home viewer B thereof via the Internet 5.

In step S453, virtual home viewer B sends a request to theinter-user-terminal communication relay server 23 via the Internet 5 forparticipating in conference (B) for user A as a chat addressed tovirtual home viewer A. Chat herein denotes a mechanism for receiving andtransmitting various data realtime or denotes the various data, whichare received and transmitted realtime.

In step S462, the inter-user-terminal communication relay server 23sends a chat addressed to virtual home viewer B (however, in thisexample, there is no chat addressed to virtual home viewer B, so thatnothing is sent).

When virtual home viewer A requests the inter-user-terminalcommunication relay server 23 via the Internet 5 for the acquisition ofa chat as a periodical operation in step S432, then theinter-user-terminal communication relay server 23 sends to virtual homeviewer A via the Internet 5 a request for participating in conference(B) for user A as a chat addressed to virtual home viewer A in stepS462.

In response to the request for participating in conference (B), in stepS433, the virtual home viewer A notifies the inter-user-terminalcommunication relay server 23 via the base station 4 and the Internet 5of the participation by user A in conference (B). In response to thisnotification, in step S463, the inter-user-terminal communication relayserver 23 adds conference (B) to user A and notifies the virtual homeviewer A thereof via the Internet 5 and the base station 4. This is theend of the description of the conference generation processing.

Subsequently, since user A and user B are participating in conference(B), messages and data can be communicated realtime between virtual homeviewer A and virtual home viewer B by means of chats. This also allowsboth the viewers to share various data (to be detailed later) assynchronous information.

Upon completion of the conference generation processing, the main panel105 of virtual home viewer A displays that the mascot 121 is visitingthe virtual home of user B, standing in front of the door thereof, asshown in FIG. 78. The control panel 106 shows “Enter Home” button 381,which is clicked to let the mascot 121 go into the virtual room of userB, “My Home” button 382, which is clicked to make the mascot 121 go backto the virtual room of user A, and “Chime” button 383, which is clickedto ring the chime of the interphone (common to the telephone) of thevirtual room of user B. It should be noted that the “Enter Home” button381 can be clicked only when user B unlocks the door of the virtualroom.

At this moment, if user B changes the layout of the external view of thevirtual room, for example, the difference between before and afterupdate of the layout information is sent also to virtual home viewer Aas the synchronous information of conference (B) (in the drawings,conference (B) is shown as Conf. (B)). The following describes theprocessing of sharing information corresponding to the operation done byuser B with reference to the flowchart shown in FIG. 79.

When user B changes the layout of the external view of the virtual room,then, in step S471, virtual home viewer B notifies theinter-user-terminal communication relay server 23 via the Internet 5 ofthe difference between before and after update of layout information asthe synchronous information of conference (B). In response to thisnotification, in step S481, the inter-user-terminal communication relayserver 23 holds the received synchronous information and notifiesvirtual home viewer B via the Internet 5 of the synchronous informationof conference (B) held at this moment.

On the other hand, if virtual home viewer A requests, as a periodicaloperation, the inter-user-terminal communication relay server 23 via thebase station 4 and the Internet 5 for the synchronous information ofconference (B) in step S491, then the inter-user-terminal communicationrelay server 23 sends the difference between before and after update ofthe layout information of the virtual room of user B to virtual homeviewer A as the synchronous information of conference (B) held at thismoment in step S482. This is the end of the description of theprocessing for sharing the information corresponding to user B'soperation.

If, as shown in FIG. 80, “Chime” button 383 is clicked by user A,virtual home viewer B is also notified thereof as the synchronousinformation of conference (B). The following describes the processingfor sharing the information corresponding to the operation by user A,with reference to the flowchart shown in FIG. 81.

When the “Chime” button 383 is clicked by user A, then, in step S501,virtual home viewer A notifies the inter-user-terminal communicationrelay server 23 via the base station 4 and the Internet 5 of the ringingof the chime of the virtual room of user B, as the synchronousinformation of conference (B). In response to this notification, in stepS511, the inter-user-terminal communication relay server 23 holds thereceived synchronous information and sends the synchronous informationof conference (B) held at this point of time to virtual home viewer Avia the Internet 5 and the base station 4.

On the other hand, if virtual home viewer B requests theinter-user-terminal communication relay server 23 via the Internet 5 forthe synchronous information of conference (B) as a periodical operationin step S521, then the inter-user-terminal communication relay server 23notifies virtual home viewer B via the Internet 5 of the ringing of thechime of the virtual room of user B as the synchronous information ofconference (B) held a this point of time. This is the end of thedescription of the processing for sharing the information correspondingto user B's operation.

The following describes processing of talking (actually, text iscommunicated realtime by means of chat) through the interphone betweenuser A and user B.

Referring to FIG. 82, there is shown a screen of virtual home viewer Bbeing displayed when the “Chime” button 383 is clicked by user A tonotify virtual home viewer B thereof as the synchronous information ofconference (B). At this moment, virtual home viewer B displays thetelephone icon 324 in the virtual room in the main panel 301 in ahighlighted manner (the icon is partially lighted or entirely blinked)and generates the sound of the beep for the interphone, which isdifferent from the beep for the telephone. Also, virtual home viewer Bdisplays, in the control panel 302, the name of one who clicked theinterphone icon and the “Answer” button 341 being clicked to answer thecall of interphone chat.

When user B clicks the “Answer” button 341, the communication panel 303has the text entry area 342 in which chat text is entered, “SEND” button343, which is clicked to send the text entered in the text entry area342, and the text display area 343.

When user B clicks the “Answer” button 341, interphone chat processingshown in FIG. 83 gets started. In step S531, virtual home viewer Bnotifies the inter-user-terminal communication relay server 23 via theInternet 5 of the answer by user B to the interphone as a chat addressedto virtual home viewer A. In step S541, the inter-user-terminalcommunication relay server 23 sends the chat addressed to virtual homeviewer B (in this example, however, the chat addressed to virtual homeviewer B does not exist, so that nothing is sent).

On the other hand, when virtual home viewer A requestsinter-user-terminal communication relay server 23 via the Internet 5 forgetting a chat as a periodical operation in step S551, then theinter-user-terminal communication relay server 23 notifies virtual homeviewer A via the Internet 5 of the answer by user B to the interphone asthe chat addressed to virtual home viewer A in step S542.

In response to this notification, virtual home viewer A switches thedisplay of the communication panel 107 to the text entry area 351 andthe text display area 353 for interphone chat as shown in FIG. 84.Subsequently, when one user sends a message, the other user receives themessage by polling, thereby enabling the interphone talk between user Aand user B (actually, texts are communicated realtime by chat).

The interphone talk between user A and user B continues until“Disconnect” button 401 shown in the control panel 302 is clicked byuser B as shown in FIG. 85.

When user A sends message “Unlock the door, please” as shown in FIG. 86,for example, user B sends a message “OK, wait a second” in response asshown in FIG. 87, user B selects the door icon 322, and clicks “Unlock”button 411 displayed in the control panel 302 upon selection of the dooricon 322, the “Enter Home” button 381 in the control panel 106 ischanged to clickable state on the screen of virtual home viewer A asshown in FIG. 88.

When user A clicks the button 381 in this state, the main panel 105 onthe screen of virtual home viewer A displays the virtual room of user Bin which the mascot 121 and the mascot 321 exist. Then, the text entryarea 351 for chat, the “SEND” button 352, and the text display area 353are arranged in the control panel 107. Likewise, the screen of virtualhome viewer B displays the virtual room of user B with the mascot 121added in the main panel 301 as shown in FIG. 90. Then, the text entryarea 342 for chat, “SEND” button 343, and the text display area 344 arearranged in the control panel 303.

The following describes the processing to be executed from unlocking toentering the room, with reference to the flowchart shown in FIG. 91.When user B clicks the “Unlock” button 411, virtual home viewer Bnotifies the inter-user-terminal communication relay server 23 via theInternet 5 of the unlocking of the virtual room of user B as thesynchronous information of conference (B) in step S561. In response tothis notification, in step S571, the inter-user-terminal communicationrelay server 23 holds the received synchronous information and sends thesynchronous information of conference (B) held at this point of time tovirtual home viewer B via the Internet 5 (in this example, thesynchronous information of conference (B) is not held, so that nothingis sent).

On the other hand, if virtual home viewer A requests theinter-user-terminal communication relay server 23 via the base station 4and the Internet 5 for the synchronous information of conference (B) asa periodical operation in step S581, then the inter-user-terminalcommunication relay server 23 notifies virtual home viewer A via theInternet 5 and the base station 4 of the unlocking of the door of theuser B's virtual room as the synchronous information of conference (B)held at this point of time in step S572.

In response to this notification, virtual home viewer A changes the“Enter Home” button 381 in the control panel 106 to the clickable state.When the “Enter Home” button 381 is clicked by user A, virtual homeviewer A notifies the inter-user-terminal communication relay server 23via the base station base station 4 and the Internet 5 of the enteringof the mascot 121 of user A into the virtual room of user B in stepS582. In response to this notification, in step S573, theinter-user-terminal communication relay server 23 sends the synchronousinformation of conference (B) held at this point of time to virtual homeviewer A via the Internet 5 and the base station 4 (in this example, thesynchronous information of conference (B) is not held, so that nothingis sent).

On the other hand, if virtual home viewer B requests theinter-user-terminal communication relay server 23 via the Internet 5 forthe synchronous information of conference (B) as a periodical operationin step S562, then the inter-user-terminal communication relay server 23notifies virtual home viewer B via the Internet 5 of the entering of themascot 121 of user A into the virtual room of user B as the synchronousinformation of conference (B) held at this point of time in step 574.

In response to this notification, in step S563, virtual home viewer Brequests the virtual home DB server 21 via the Internet 5 for the imagedata of the mascot 121 of user A (hereafter also referred to as mascotA). In response to this request, in step S591, the virtual home DBserver 21 supplies the image data of mascot A to virtual home viewer Bvia the Internet 5. Consequently, the virtual room on the screen ofvirtual home viewer B also displays mascot A.

In step S564, virtual home viewer B sends the layout information of userB's virtual room in which mascot A has entered to theinter-user-terminal communication relay server 23 as the synchronousinformation of conference (B) via the Internet 5 (the difference inlayout information may be sent instead). In response, in step S575, theinter-user-terminal communication relay server 23 holds the receivedsynchronous information and sends the synchronous information ofconference (B) held at this point of time to virtual home viewer B viathe Internet 5 (in this example, the synchronous information ofconference (B) is not held, so that nothing is sent).

On the other hand, in step S583, if virtual home viewer A requests theinter-user-terminal communication relay server 23 via the base station 4and the Internet 5 for the synchronous information of conference (B) asa periodical operation, then the inter-user-terminal communication relayserver 23 sends the layout information of user B's virtual room in whichmascot A has entered to virtual home viewer A via the Internet 5 andbase station 4 as the synchronous information of conference (B) in stepS576. Consequently, the main panel 105 of virtual home viewer A displaysthe user B's virtual room with mascot A entered. This is the end of thedescription of the processing to be executed from unlocking to enteringthe room.

The following describes the processing for one user to present an item(for example, the potted plant icon 129) to an other user. There are twomethods of presenting items; one in which an item is directly presentedbetween mascots and the other in which an item is indirectly presentedthrough the mailbox arranged on the door.

The following describes an operation for directly passing the pottedplant icon 129 from mascot 121 (mascot A) to the mascot 321 (hereafteralso referred to as mascot B).

Shown in FIG. 92, user A chooses the mascot 121 (mascot A) shown in themain panel 105 and drags and drops the potted plant icon 129 shown in apopped up balloon 421 onto mascot B. At this moment, user A cannot dropthe potted plant icon 129 onto other than mascot B (for example, thefloor).

In the balloon 421, the potted plant icon 129 is shown in reduced size;when the potted plant icon 129 is taken out of the balloon 421, the icon129 gains its original size as shown in FIG. 93.

It should be noted that the screen of virtual home viewer B shows ananimation in which mascot A passes the potted plant icon 129 to mascot Bas shown in FIG. 94. If the virtual room of user B has the mascot of another user (for example, user C), the screen of user C also shows theanimation in which mascot A passes the potted plant icon 129 to mascotB.

The following describes ownership transfer processing in which an item(in this example, the potted plant icon 129) has been directly passedfrom mascot A to mascot B, with reference to the flowchart shown in FIG.95. This direct ownership transfer processing starts upon dropping ofthe potted plant icon 129 onto mascot B on the screen of virtual homeviewer A.

In step S601, virtual home viewer A request the virtual home DB server21 via the base station 4 and the Internet 5 for an icon transfercertificate (A→B), which certifies the transfer of the ownership of thepotted plant icon 129 from user A to user B. In response to thisrequest, the virtual home DB server 21 issues the icon transfercertificate (A→B) to virtual home viewer A via the Internet 5 in stepS611.

In step S602, virtual home viewer A sends the icon transfer certificate(A→B) to the inter-user-terminal communication relay server 23 via thebase station 4 and the Internet 5 as the synchronous information ofconference (B). In response, in step S621, the inter-user-terminalcommunication relay server 23 holds the received synchronous informationand sends the synchronous information of conference (B) held at thispoint of time to virtual home viewer B via the Internet 5 (in thisexample, the synchronous information of conference (B) is not held, sothat nothing is sent).

On the other hand, if virtual home viewer B requests theinter-user-terminal communication relay server 23 via the Internet 5 forthe synchronous information of conference (B) in step S631, then theinter-user-terminal communication relay server 23 sends the icontransfer certificate (A→B) to the virtual home viewer B via the Internet5 as the synchronous information of conference (B) held at this point oftime in step S622.

In step S632, virtual home viewer B sends the icon transfer certificate(A→B) to the virtual home DB server 21 via the Internet 5, therebyrequesting for the transfer of the ownership of the potted plant icon129. In response to this request, in step S612, the virtual home DBserver 21 transfers the ownership of the potted plant icon 129 to user Band notifies virtual home viewer B thereof via the Internet 5. This isthe end of the description of the direct ownership transfer processing.

Upon completion of the direct ownership transfer processing, if user Bchooses the mascot 321 (mascot B) in the virtual room shown in the mainpanel 301 as shown in FIG. 96, the balloon 431 in which an item carriedby mascot 321 is shown is popped up, so that user B can makeconfirmation of the item presented by user A.

Further, dragging and dropping the item (in this example, the pottedplant icon 129) shown in the popped up balloon 431 onto a desiredposition in the virtual room allow the presented item to be placed asdesired in its original size as shown in FIG. 97.

In order to make a present from user B to user A when mascot A is in thevirtual room of user B, user B may drag and drop the item to bepresented onto mascot A.

The following describes the method of indirectly passing an item throughthe mailbox arranged on the door. First, an operation will be describedin which the memo sheet icon 241 carried by mascot A out of the room ispassed to mascot B via the mailbox arranged on the door.

User A chooses mascot A displayed in the main panel 105 to pop up theballoon 451 as shown in FIG. 98 and drags and drops the memo sheet icon241 shown in the balloon 451 onto the mailbox icon 323 arranged on theoutside of the door as shown in FIG. 99.

In response to this operation, on the screen of virtual home viewer B,as shown in FIG. 100, the external view of the mailbox icon 323 ischanged to the state showing that the item is in the mailbox icon 323,in the virtual room displayed on the main panel 301. When user B clicksthe mailbox icon 323, the balloon icon 461 pops up, in which the item(in this example, the memo sheet icon 241) passed from user A isdisplayed. Choosing the mailbox icon 323 causes the control panel 302 todisplay the number of items in the mailbox, for example, as the propertyof the mailbox icon 323.

Further, when user B drags and drops the memo sheet icon 241 shown inthe balloon 461 onto a desired place in the virtual room, the item (inthis example, the memo sheet icon 241) passed through the mailbox can bearranged at a desired place in its original size in the virtual room asshown in FIG. 101.

The following describes the ownership transfer processing in which anitem (in this example, the memo sheet icon 241) is indirectly passed,with reference to the flowchart shown in FIG. 102. This indirectownership transfer processing starts when the memo sheet icon 241 isdragged and dropped on the mailbox icon 323 outside the door on thescreen of virtual home viewer A.

In step S641, virtual home viewer A requests the virtual home DB server21 via the base station 4 and Internet 5 for the icon transfercertificate (A→B), which certifies the transfer of the ownership of thememo sheet icon 241 from user A to user B. In response to this request,in step S651, the virtual home DB server 21 issues the icon transfercertificate (A→B) to virtual home viewer A via the Internet 5.

In step S642, virtual home viewer A sends the icon transfer certificate(A→B) to the user information holding server 24 via the base station 4and the Internet 5, thereby requesting for its storage as an object. Inresponse to this request, in step S661, the user information holdingserver 24 sends the icon transfer certificate (A→B) and stores thecertificate as an object, thereby notifying virtual home viewer Athereof via the Internet 5.

In step S643, virtual home viewer A notifies the virtual home DB server21 via the base station 4 and the Internet 5 of an event informationindicative that the item has been delivered to user B. In response tothis notification, in step S652, the virtual home DB server 21 holds theevent information indicative that the item has been delivered to user Band notifies virtual home viewer A thereof via the Internet 5.

On the other hand, if virtual home viewer B requests the virtual home DBserver 21 via the Internet 5 for the event information as a periodicaloperation in step S671, then the virtual home DB server 21 sends theevent information indicative that the item has been delivered to user Bto virtual home viewer B via the Internet 5 in step S653. In response,on the screen of virtual home viewer B, as shown in FIG. 100, theexternal view of the mailbox icon 323 is changed to the state in whichsomething is in the mailbox in the virtual room displayed on the mainpanel 301.

In step S672, virtual home viewer B requests the user informationholding server 24 via the Internet 5 for icon transfer certificate (A→B)held in this server by user A. In response to this request, in stepS662, the user information holding server 24 supplies the icon transfercertificate (A→B) to virtual home viewer B via the Internet 5.

In step S673, virtual home viewer B requests the virtual home DB server21 via the Internet 5 for the deletion of the event informationindicative that the item has been delivered to user B. In response tothis request, in step S654, the virtual home DB server 21 deletes theevent information indicative that the item has been delivered to user B.

In step S674, virtual home viewer B requests the user informationholding server 24 via the Internet 5 to delete the icon transfercertificate (A→B), which is held in this server by user A. In responseto this request, in step S663, the user information holding server 24deletes the icon transfer certificate (A→B) and notifies virtual homeviewer B thereof via the Internet 5.

In step S675, virtual home viewer B sends the icon transfer certificate(A→B) to the virtual home DB server 21 via the Internet 5 to request forthe transfer of the ownership of the item (in this case, the memo sheeticon 241). In response to this request, in step S655, the virtual homeDB server 21 transfers the ownership of the memo sheet icon 241 andnotifies virtual home viewer B thereof via the Internet 5. This is theend of the description of the indirect ownership transfer processing.

Upon completion of the indirect ownership transfer processing, user Bchooses the mailbox icon 323 in the virtual room displayed on the mainpanel 301 to pop up the balloon 461, by which user B can makeconfirmation of the delivered item.

Further, dragging and dropping the item (in this example, the memo sheeticon 241) shown in the popped up balloon 461 onto a desired place in thevirtual room can arrange the delivered item in its original size at anydesired place.

The following describes BGM (BackGround Music) sharing processing forreproducing same BGM between virtual home viewer A operated by user Aand virtual home viewer B operated by user B with the mascot 321 (mascotB) of user B visiting the virtual room of user A, for example. This BGMsharing processing uses the stream data of so-called Internet radiodistributed by the audio server 26.

Referring to FIG. 103, there is shown the screen of virtual home viewerA, in which the main panel 105 shows mascot B visiting the virtual roomof user A. It should be noted that, although not shown, the same imageis shown in the main panel 301 on the screen of virtual home viewer B.

When user A chooses the audio player icon 172 of the virtual room shownin the main panel 105, the control panel 106 displays, as the propertyof the audio player icon 172, an information display area 471 fordisplaying the information about the BGM being reproduced, “POWER”button 472, which is clicked to turn on/off the output of BGM, and“Channel” button 473, which is clicked to change BGM.

Further, as shown in FIG. 104, when user A clicks the “POWER” button472, the output of BGM is turned on, starting the reception andreproduction of the stream data distributed from the audio server 26.Likewise, in the virtual home viewer B, the reception and reproductionof the same stream data distributed from the audio server 26 arestarted.

The following describes the BGM sharing processing, which is startedupon clicking of the “POWER” button 472 by user A, with reference to theflowchart shown in FIG. 105.

In step S681, virtual home viewer A requests the audio server 26 via thebase station 4 and the Internet 5 for stream reproduction (requests forthe distribution of stream data). In step S691, the audio server 26notifies virtual home viewer A via the Internet 5 and the base station 4of the acknowledgement of this request and, in step S692, startsdistributing the stream data to virtual home viewer A.

Consequently, virtual home viewer A starts reception and reproduction ofthe stream data and outputs BGM. In step S682, virtual home viewer Anotifies the inter-user-terminal communication relay server 23 via thebase station 4 and the Internet 5 of the turning-on of the output of BGMas the synchronous information of conference (B).

On the other hand, if, in step S711, virtual home viewer B requests theinter-user-terminal communication relay server 23 via the Internet 5 forthe synchronous information of conference (B) as a periodical operation,then the inter-user-terminal communication relay server 23 notifiesvirtual home viewer B via the Internet 5 of the turning-on of the outputof BGM as the synchronous information of conference (B) held at thispoint of time.

In response to this notification, in step S712, virtual home viewer Brequests the audio server 26 via the Internet 5 for stream reproduction(requests for the distribution of stream data). In step S693, the audioserver 26 notifies virtual home viewer B via the Internet 5 of theacknowledgement of this request and starts distributing the stream datato virtual home viewer B in step S694. Consequently, virtual home viewerB also starts reception and reproduction of the stream data and outputsBGM. This is the end of the description of the BGM sharing processing.

It should be noted that, in the BGM sharing processing, the music dataaccumulated in the equipment on the user side (for example, the PC 12)may be stream-distributed to virtual home viewer A or virtual homeviewer B, rather than using the Internet radio stream data to bedistributed by the audio server 26 on the service provider side.

The following describes image sharing processing for displaying theimage data held by user A onto the screen of virtual home viewer Aoperated by user A and the screen of virtual home viewer B operated byuser B with the mascot 321 (mascot B) of user B visiting the virtualroom of user A, for example. This image sharing processing uses theimage data managed by the photograph data transmission service 94, whichis a functional block of the PC 12, for example.

Referring to FIG. 106, there is shown the screen of virtual home viewerA, in which the visiting of the virtual room of user A by mascot B isshown in the main panel 105. It should be noted that, although notshown, the same image is shown in the main panel 301 on the screen ofvirtual home viewer B.

When user A chooses the photo album icon 173 in the virtual room shownin the main panel 105, the control panel 106 displays, as the propertyof the photo album icon 173, an information display area 481 in whichthe information about image data is displayed, “Display Image List”button 482, which is clicked when displaying an image list including twoor more reduced-size images onto the main panel 105, and “Close” button483, which is clicked when returning the display of the main panel 105from the image list to the virtual room.

If user A clicks the “Display Image List” button 482, the main panel 105displays an image list including reduced-size images corresponding tothe image data managed by the photograph data transmission service 94 asshown in FIG. 107.

When user A selects one of the images from the image list shown in themain panel 105, the information display area 481 displays the image sizeand date of creation of the selected reduced-size image as shown in FIG.108.

The control panel 106 also displays “Zoom In” button 491 being clickedto zoom in a selected reduced-size image into the main panel 105,“Display to All” button 492 being clicked to share the image data of theselected reduced-size image to all users in the virtual room, “DisplayList” button 493 being clicked to return the display of the main panel105 from the zoomed-in image to the image list, “Display to All” button494 being clicked to delete the selected reduced-size image from theimage list, and “Close” button 495 being clicked to return the displayof the main panel 105 from the image list to the virtual room.

If user A clicks the “Display to All” button 492, the selectedreduced-size image is displayed in the main panel 105 as zoomed in asshown in FIG. 109 and the image data corresponding to the selectedreduced-size image are supplied to the other user (in this example, userB).

The following describes the image sharing processing being started byclicking the “Display to All” button 492, with reference to theflowchart shown in FIG. 110.

In step S721, virtual home viewer A sends the shared image data to theuser information holding server 24 via the Internet 5, requesting forthe storage of the sent data. In response to this request, in step S731,the user information holding server 24 holds the received shared imagedata and notifies virtual home viewer A thereof via the Internet 5.

In step S722, virtual home viewer A issues an image acquisition permit(A→B), which permits the sharing of the image data of user A with user Band sends the image acquisition permit (A→B), to the inter-user-terminalcommunication relay server 23 via the Internet 5 as the synchronousinformation of conference (B) In response, in step S741, theinter-user-terminal communication relay server 23 holds the receivedsynchronous information and sends the synchronous information ofconference (B) held at this point of time to virtual home viewer A viathe Internet 5 (in this example, the synchronous information ofconference (B) is not held, so that nothing is sent).

On the other hand, if virtual home viewer B requests theinter-user-terminal communication relay server 23 via the Internet 5 forthe synchronous information of conference (B) as a periodical operationin step S751, then the inter-user-terminal communication relay server 23sends the image acquisition permit (A→B) to virtual home viewer B viathe Internet 5 as the synchronous information of conference (B) held atthis point of time in step S742.

In step S752, virtual home viewer B sends the image acquisition permit(A→B) to the user information holding server 24 via the Internet 5,requesting for the supply of the image data. In response to thisrequest, in step S732, the user information holding server 24 suppliesthe image data stored by the processing of step S731 to virtual homeviewer B via the Internet 5. Virtual home viewer B displays the suppliedimage data in the main panel 301 as shown in FIG. 111. Consequently, thesame image is displayed in both the main panel 105 of virtual homeviewer A and the main panel 301 of virtual home viewer B. This is theend of the description of the image sharing processing.

If user A clicks the “Display List” button 493 or the “Close” button 495with the same image displayed in both the main panel 105 of virtual homeviewer A and the main panel 301 of virtual home viewer B by theabove-mentioned image sharing processing, image sharing end processingis executed, upon which the display of the main panel 301 of virtualhome viewer B is returned to the virtual room of user A.

The following describes this image sharing end processing with referenceto the flowchart shown in FIG. 112. In step S761, virtual home viewer Anotifies the inter-user-terminal communication relay server 23 via theInternet 5 of the end of browsing of shared image data as thesynchronous information of conference (B). In response to thisnotification, in step S771, the inter-user-terminal communication relayserver 23 holds the received synchronous information and sends thesynchronous information of conference (B) held at this point of time tovirtual home viewer A via the Internet 5 (in this example, thesynchronous information of conference (B) is not held, so that nothingis sent).

On the other hand, if virtual home viewer B requests theinter-user-terminal communication relay server 23 for the synchronousinformation of conference (B) as a periodical operation in step S781,then the inter-user-terminal communication relay server 23 notifiesvirtual home viewer B of the end of shared browsing of the image data asthe synchronous information of conference (B) held at this point of timein step S772. In response to this notification, virtual home viewer Breturns the display of the main panel 301 to the virtual room of user A.

In step S762, virtual home viewer A requests the user informationholding server 24 via the Internet 5 for the deletion of the image dataheld for shared use. In response to this request, in step S791, the userinformation holding server 24 deletes the image data stored therein andnotifies virtual home viewer A thereof via the Internet 5. This is theend of the description of the image sharing end processing.

As described and according to the virtual space remote control systemassociated with the present invention, not only the household electricalappliances connected to the home network can be controlled by theentertainment-oriented GUI such as the virtual room, but also therealtime message exchange (namely telephone chat and interphone chat),the sharing of the virtual room by plural users, the sharing of BGM, andthe sharing of image data can be performed.

As described above, the entertainment-oriented andcommunication-oriented GUI can increase the number of users who use thevirtual space remote control system according to the invention. Thesefeatures of the invention can also increase the frequency of usage ofthe virtual space remote control system according to the invention.

If, in order to increase in the number of icons arranged in the virtualroom, the user purchases electronic devices and connect the devices tothe home network, it results in the increase in the number of devices,which can be controlled by the virtual home viewer. This creates apositive spiral that the increased frequency of use in turn increasesthe number of electronic devices.

Use of the capabilities of visiting the virtual rooms of other usersallows the establishment of a model room for advertising the features ofnetwork devices and network devices themselves. Also, use of chats inthis model room allows the advertisement of commercial products.

The steps herein for describing the program recorded to the recordingmedium include not only the processing operations being executed in atime-dependent manner in accordance with the described sequence, butalso processing operations being executed not always in a time-dependentmanner but in a parallel or discrete manner.

INDUSTRIAL APPLICABILITY

As described and according to an embodiment of the present invention, avirtual space can be shared between a plurality of users to exchangemessages and share image data.

In addition, according to an embodiment of the present invention, a userinterface having an entertainment-oriented virtual space can be used tocontrol the electrical household appliances connected to a home network.

1. An information processing system, comprising: a plurality ofinformation processing apparatuses; and a service providing apparatusconnected with the plurality of information processing apparatuses overa network; the service providing apparatus including: providing meansfor providing, to a given one of the plurality of information processingapparatuses, a service of a private virtual space in which a firstmascot and an icon are arranged, wherein the private virtual space, thefirst mascot and the icon are associated with a particular user of theplurality of information processing apparatuses, the icon representingan actual electrical device that is located in a real space associatedwith the particular user and that is connected over a local network to aparticular one of the plurality of information processing apparatuses,storage means for storing layout information defining a layout of theprivate virtual space and for storing a selected location of the iconwithin the layout of the private virtual space, the location of the iconbeing selected by the particular user, and sharing means for sharing theservice of the private virtual space being provided to the given one ofthe plurality of information processing apparatuses with another one ofthe plurality of information processing apparatuses being used byanother user; the given one of the plurality of information processingapparatuses and the another one of the plurality of informationprocessing apparatuses each including: display control means forcontrolling display of the private virtual space in which the icon isarranged, and in response to the icon being selected by the particularuser or the another user, for controlling display of the state of theelectrical device and display of controls by which an operation isperformed by the particular user or by the another user, wherein theactual electrical device is a television set, the icon arranged in theprivate virtual space is a television icon, and said controlling displayof the state of the electrical device by said display control meansincludes modifying the television icon to display the channel currentlybeing received by the television set when the television set is in apower-on state; the particular one of the plurality of informationprocessing apparatuses including: control means for controlling theelectrical device over the local network in response to a the operationperformed by the particular user or by the another user.
 2. Theinformation processing system according to claim 1, wherein the serviceproviding apparatus is connected with the plurality of informationprocessing apparatuses over the Internet.
 3. The information processingsystem according to claim 1, wherein the local network is a homenetwork, and the electrical device is located in the home of theparticular user.
 4. The information processing system according to claim1, wherein the local network is a home network, and the particular oneof the plurality of information processing apparatuses is located in thehome of the particular user.
 5. The information processing systemaccording to claim 1, wherein the operation is performed by theparticular user using the given one of plurality of informationprocessing apparatuses, and the sharing means shares the operationperformed by the particular user with the particular one of theplurality of information processing apparatuses.
 6. The informationprocessing system according to claim 1, wherein the operation isperformed by the another user using the another one of the plurality ofinformation processing apparatuses, and the sharing means shares theoperation performed by the another user with the another one of theplurality of information processing apparatuses.
 7. The informationprocessing system according to claim 1, wherein the given one of theplurality of information processing apparatuses and the another one ofthe plurality of information processing apparatuses each include:request means for sending, to the service providing apparatus, theoperation performed by the particular user or by the another user. 8.The information processing system according to claim 1, wherein thegiven one of the plurality of information processing apparatuses and theanother one of the plurality of information processing apparatuses eachinclude: display control means for controlling the displaying of theprivate virtual space, the private virtual space being provided by thesharing means of the service providing apparatus.
 9. A service providingapparatus connected with a plurality of information processingapparatuses over a network, the service providing apparatus comprising:providing means for providing, to a given one of the plurality ofinformation processing apparatuses, a service of a private virtual spacein which a first mascot and an icon are arranged, wherein the privatevirtual space, the first mascot and the icon are associated with aparticular user of the plurality of information processing apparatuses,the icon representing an actual electrical device that is located in areal space associated with the particular user and that is connectedover a local network to a particular one of the plurality of informationprocessing apparatuses; the providing means for providing, to the givenone of the plurality of information processing apparatuses in responseto the icon being selected by the particular user, the state of theelectrical device and controls by which an operation is performed by theparticular user or by the another user; storage means for storing layoutinformation defining a layout of the private virtual space and forstoring a selected location of the icon within the layout of the privatevirtual space, the location of the icon being selected by the particularuser; and sharing means for sharing the service of the private virtualspace being provided to the given one of the plurality of informationprocessing apparatuses with another one of the plurality of informationprocessing apparatuses being used by another user, and in response tothe icon being selected by the another user, for sharing the state ofthe electrical device and controls by which the operation is performedby the another user, wherein the actual electrical device is atelevision set, the icon arranged in the private virtual space is atelevision icon, and said sharing the state of the electrical device bysaid sharing means includes sharing within the television icon displayof the channel currently being received by the television set when thetelevision set is in a power-on state; the sharing means for sharing,with the particular one of the plurality of information processingapparatuses, the operation performed by the particular user or by theanother user which, in response thereto, the particular one of theplurality of information processing apparatuses controls the electricaldevice.
 10. The service providing apparatus according to claim 9,wherein the service providing apparatus is connected with the pluralityof information processing apparatuses over the Internet.
 11. The serviceproviding apparatus according to claim 9, wherein the local network is ahome network, and the electrical device is located in the home of theparticular user.
 12. The service providing apparatus according to claim9, wherein the local network is a home network, and the particular oneof the plurality of information processing apparatuses is located in thehome of the particular user.
 13. The service providing apparatusaccording to claim 9, wherein the selection of the icon is performed bythe particular user using the given one of plurality of informationprocessing apparatuses or by the another user using the another one ofthe plurality of information processing apparatuses.
 14. The serviceproviding apparatus according to claim 9, wherein the operation isperformed by the particular user using the given one of plurality ofinformation processing apparatuses or by the another user using theanother one of the plurality of information processing apparatuses. 15.An information processing method, comprising: providing, to a given oneof a plurality of information processing apparatuses using a serviceproviding apparatus connected with the plurality of informationprocessing apparatuses over a network, a service of a private virtualspace in which a first mascot and an icon are arranged, wherein theprivate virtual space, the first mascot and the icon are associated witha particular user of the plurality of information processingapparatuses, the icon representing an actual electrical device that islocated in a real space associated with the particular user and that isconnected over a local network to a particular one of the plurality ofinformation processing apparatuses; storing, at the service providingapparatus, layout information defining a layout of the private virtualspace and storing a selected location of the icon within the layout ofthe private virtual space, the location of the icon being selected bythe particular user; controlling, at the given one of the plurality ofinformation processing apparatuses, display of the private virtual spacein which the icon is arranged; in response to the icon being selected bythe particular user at the given one of the plurality of informationprocessing apparatuses, controlling display of the state of theelectrical device and display of controls by which an operation isperformed by the particular user, wherein the electrical device is atelevision set, the icon arranged in the private virtual space is atelevision icon, and said controlling display of the state of theelectrical device includes modifying the television icon to display thechannel currently being received by the television set when thetelevision set is in a power-on state; sharing, at the service providingapparatus, the service of the private virtual space being provided tothe given one of the plurality of information processing apparatuseswith another one of the plurality of information processing apparatusesbeing used by another user; controlling, at the given one of theplurality of information processing apparatuses or the another one ofthe plurality of information processing apparatuses, display of theprivate virtual space in which the icon is arranged; in response to theicon being selected by the another user at the another one of theplurality of information processing apparatuses, controlling display ofthe state of the electrical device and display of the controls by whichthe operation is performed by the another user; and controlling, usingthe particular one of the plurality of information processingapparatuses, the electrical device in response to the operationperformed by the particular user or by the another user.
 16. A serviceproviding method, comprising: providing, over a network to a given oneof a plurality of information processing apparatuses, a service of aprivate virtual space in which a first mascot and an icon are arranged,wherein the private virtual space, the first mascot and the icon areassociated with a particular user of the plurality of informationprocessing apparatuses, the icon representing an actual electricaldevice that is located in a real space associated with the particularuser and that is connected over a local network to a particular one ofthe plurality of information processing apparatuses; providing, to thegiven one of the plurality of information processing apparatuses inresponse to the icon being selected by the particular user, the state ofthe electrical device and controls by which an operation is performed bythe particular user or by the another user; storing layout informationdefining a layout of the private virtual space and storing a selectedlocation of the icon within the layout of the private virtual space, thelocation of the icon being selected by the particular user; sharing,over the network with another one of the plurality of informationprocessing apparatuses used by another user, the service of the privatevirtual space being provided to the given one of the plurality ofinformation processing apparatuses; sharing, over the network with theanother one of the plurality of information processing apparatuses inresponse to the icon being selected by the another user, the state ofthe electrical device and the controls by which the operation isperformed by the another user, wherein the actual electrical device is atelevision set, the icon arranged in the private virtual space is atelevision icon, and said sharing the state of the electrical deviceincludes sharing within the television icon display of the channelcurrently being received by the television set when the television setis in a power-on state; and sharing, over the network with theparticular one of the plurality of information processing apparatuses,the operation performed by the particular user using the given one ofplurality of information processing apparatuses or by the another userusing the another one of the plurality of information processingapparatuses which, in response thereto, the particular one of theplurality of information processing apparatuses controls the electricaldevice.
 17. The information processing system according to claim 1,wherein said display control means, in response to the operationperformed by the particular user or by the another user, changes thedisplay of the state of the electrical device based on the operationperformed.
 18. The information processing system according to claim 17,wherein the operation performed by the particular user or by the anotheruser includes changing the channel currently being received by thetelevision set from a first channel to a second channel, and the changeof the display of the state of the television set by the display controlmeans includes modifying the television icon from displaying the firstchannel to displaying the second channel.
 19. The service providingapparatus according to claim 9, wherein said sharing means, in responseto the operation performed by the particular user or by the anotheruser, changes the shared display of the state of the electrical devicebased on the operation performed.
 20. The service providing apparatusaccording to claim 19, wherein the operation performed by the particularuser or by the another user includes changing the channel currentlybeing received by the television set from a first channel to a secondchannel, said sharing means modifying the shared television icon fromdisplaying the first channel to displaying the second channel.
 21. Theinformation processing method according to claim 15, further comprising:changing, in response to the operation performed by the particular useror by the another user, the display of the state of the electricaldevice based on the operation performed.
 22. The information processingmethod according to claim 21, wherein the operation performed by theparticular user or by the another user includes changing the channelcurrently being received by the television set from a first channel to asecond channel, and said changing the display of the state of theelectrical device includes modifying the television icon from displayingthe first channel to displaying the second channel.
 23. The serviceproviding method according to claim 16, further comprising: changing, inresponse to the operation performed by the particular user or by theanother user, the shared display of the state of the electrical devicebased on the operation performed.
 24. The service providing methodaccording to claim 23, wherein the operation performed by the particularuser or by the another user includes changing the channel currentlybeing received by the television set from a first channel to a secondchannel, and said changing the shared display of the state of theelectrical device includes modifying the television icon from displayingthe first channel to displaying the second channel.